[Tcl-bugs] [ tcl-Bugs-1578344 ] -exact version matching broken

[SourceForge.net <no...@so...>]

Bugs item #1578344, was opened at 2006-10-16 10:10
Message generated for change (Comment added) made by andreas_kupries
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=1578344&group_id=10894

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: 38. Package Manager
Group: obsolete: 8.5a5
Status: Open
Resolution: None
Priority: 8
Private: No
Submitted By: Don Porter (dgp)
Assigned to: Joe English (jenglish)
Summary: -exact version matching broken

Initial Comment:

The TIP 268 implementation
broke this:

% package provide demo 1.2.3
% package require -exact demo 1.2    
1.2.3

Contrast with Tcl 8.4:

% package provide demo 1.2.3
% package require -exact demo 1.2    
version conflict for package "demo": have 1.2.3, need 1.2
% info patch
8.4.14

And with what I recall being
specified in TIP 268.



----------------------------------------------------------------------

>Comment By: Andreas Kupries (andreas_kupries)
Date: 2007-05-30 13:34

Message:
Logged In: YES 
user_id=75003
Originator: NO

Reading over your changes to AddRequirementsToResult, do I understand
correctly that you are translating the form 'V-V' back into the old-style
form, i.e. '-exact V' ? If yes, is the 'exactly' not a typo ?

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-29 08:56

Message:
Logged In: YES 
user_id=80530
Originator: YES


I think these patches do the right thing.

They provide compatible support for
the sensible historic ways to call
Tcl_InitStubs():

  Tcl_InitStubs(i, TCL_VERSION, 1)

This would be called to get a Tcl stubs
table from the same TCL_VERSION series
as was used to compile the extension.
With this patch, that same meaning is
maintained, without the need to damage
the compatibility of what 
[package require -exact] has always meant.

  Tcl_InitStubs(i, TCL_VERSION, 0)

would get a stub table from the same
TCL_VERSION series as was used to
compile the extension, or from any
compatible later version.  Again this
patch does the same thing when
called in the same way.

In addition, this patch permits some
other cases identified by jenglish to
be addressed for the first time.

  Tcl_InitStubs(i, TCL_PATCH_LEVEL, 0)

can be used to set a more discriminating
floor on the set of acceptable stub tables,
so that, say, stub tables from only Tcl 8.5.2
and later would be accepted, and attempts to
call in a Tcl 8.5.0 interp would fail.

  Tcl_InitStubs(i, TCL_PATCH_LEVEL, 1)

could be used to demand lockstep requirement
for a stub table from a single release of Tcl.
I tend to dismiss this a lot, since it counters
the multiple version capabilities provided by
stubs.  But there can be other reasons to use stubs
(getting an architecture suitable for starkit inclusion,
for instance), and from a strict perspective of
"support only what I test", this can be a useful
tool for some developers and some needs.

There are some other more complex requirement
needs that cannot be expressed with a single
Tcl_InitStubs() call even with this
patch in place, but I really cannot imagine
completely addressing those without an entirely
new interface routine.

There more complex requirements *can* be resolved
with this patch, if one uses Tcl_InitStubs() to
first get an 8.5a7 stubs table, and then make
a call to Tcl_PkgRequireProc() to make a requirements
test with the full power and generality of the
TIP 268 enhanced [package require] command. 

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-29 06:34

Message:
Logged In: YES 
user_id=80530
Originator: YES


Updated patch takes care of a few more
details.
File Added: 1578344-3.patch

----------------------------------------------------------------------

Comment By: Joe English (jenglish)
Date: 2007-05-23 13:08

Message:
Logged In: YES 
user_id=68433
Originator: NO

I don't think this is the right way to fix the problem.

Instead of changing the semantics of Tcl_InitStubs() to be what Tk wants
it to mean, Tk 8.5 should instead say exactly what it means (possibly by
calling Tcl_InitStubs(interp, TCL_VERSION, 0); to initialize the stubs
table, followed by further calls to Tcl_PkgRequirePrefixMatch to express
the exact requirement).

Extensions other than Tk might very well want Tcl_InitStubs(interp,
TCL_PATCH_LEVEL, 1); to mean "the exact version that I was compiled
against, no prefix matches".  In particular, extensions that reference
private headers might want this behaviour.

(That's been a problem with the 8.4 series -- there was no way to say "I
need Tcl 8.4.N or later, because I'm using a routine that was added to the
internal stubs table in that version", and there was no way to say "I want
Tcl 8.4.N exactly, because ditto and furthermore there's no guarantee that
it will still be in Tcl 8.4.N+1").

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-23 11:00

Message:
Logged In: YES 
user_id=80530
Originator: YES

attached patch is a small tweak
on the first one that I think
corrects the failure mode.

jenglish, care to have another look?


File Added: 1578344-2.patch

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-23 08:44

Message:
Logged In: YES 
user_id=80530
Originator: YES


jenglish points out that
the changes to Tcl_InitStubs()
in that patch won't work in 
all the cases they need to work.

If this patch went into Tcl 8.5,
and then an extension was compiled
against 8.5 headers, and linked to
libtclstub8.5a, that extension's
shared library file would contain
the modified Tcl_InitStubs() routine.

If that shared library file were
[load]ed into a Tcl 8.3 interp
(and the prefixMatching argument
were true), then an attempt to call
Tcl_PkgRequirePrefixMatching through
the stubs table would be attempted,
and since no such routine is in
an 8.3 stubs table the program would
crash.

back to that drawing board...


----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-22 11:26

Message:
Logged In: YES 
user_id=80530
Originator: YES


The attached patch corrects everything
except the docs.

It corrects the bug reported here, but
also restores the compatibility of
Tcl_InitStubs() that was lost with the
original TIP 268 implementation, which
prompted the DWIM redefine for -exact
matching.  I think this solution ought
to satisfy everyone.

A backport of this solution to the
TCL_TIP268 sections on the core-8-4-branch
might be a good idea as well, but it's best
I leave that conclustion to the actual users
of that fork.  I'll help with producing such
a patch if it's desired.

File Added: 1578344.patch

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-05-18 13:12

Message:
Logged In: YES 
user_id=80530
Originator: YES


Ok, I see (at least some of) the
motivation for this now.  It comes
down to a deficiency in the Tcl_InitStubs()
interface.

Consider Tk 8.4 as an example of a stubs
enabled extension.  It calls

  Tcl_InitStubs(interp, TCL_VERSION, 1)

The Tk header file forces TCL_VERSION
to be "8.4", so this call is looking to
initialize "exactly" version 8.4 of
the Tcl stub table.

Trouble is that the stub table itself
does not include any version field.
Instead the "version" of a stub table
is implicitly taken to be the same
as the version of the "Tcl" package
in the same interp, via a call in
Tcl_InitStubs:

  Tcl_PkgRequireEx(interp, "Tcl",
      version, exact, &pkgData);

where version and exact have the
values TCL_VERSION and 1 passed into
Tcl_InitStubs originally.

Since the history has been that
Tcl only [package provide]d a
Tcl version to M.m resolution, the
implicit rule is that the Tcl stub
table was version 8.4 throughout
the entire 8.4* series of releases.

This has a few implications.  First
a value of 1 in the "exact" argument
to Tcl_InitStubs() has always meant
that any stub table from any 8.4*
release of Tcl is acceptable, but
that any table from 8.3* or 8.5*
releases is not.  For extensions like
Tk that may be accessing volatile internals,
this is a good kind of requirement to be
able to express.

Note that the history of having a single
version identifier for all the stub tables
of all Tcl releases matching 8.4.*
indicates that the stub table is meant
to be unchanging through the whole
8.4* series of releases.  Except for the
backport of TIP 218, we've even honored
that (in the default build anyway).

After TIP 268, [package provide Tcl]
now returns the full patchlevel version
number.  Since the workings of
Tcl_InitStubs() were not modified by
that TIP, we now effectively have
a new version of the stub table with
every release of Tcl.  This means
that Tcl_InitStubs() calls with exact=1
can only match a single release of Tcl,
which is not desirable.  Furthermore
passing both TCL_VERSION and exact=1
can only match the stub table to be
found in Tcl 8.5.0 which does not yet
exist.  This means without source code
changes, Tk 8.5a6 could not [load] into
any existing Tcl interp.

Tk 8.5a6 has addressed this in part
by changing its Tcl_InitStubs call to:

  Tcl_InitStubs(interp, TCL_PATCH_LEVEL, 1)

With the traditional meaning of -exact
matching, this would permit a Tk 8.5a6
built against Tcl 8.5a6 header file to [load] into
a Tcl 8.5a6 interp, but no other release at all.
This level of lockstep forced even on the
stubs interface is undesirable.

The old ability of Tcl_InitStubs to
approve the stub table of all Tcl
releases match 8.5* was restored by
the incomptible interpretation of
-exact matching noted in the original
bug report here.  It's an example
of restoring one kind of compatibility
by destroying another kind.

I think for the 8.5 series of releases,
this issue can successfully be solved
by making appropriate changes to
Tcl_InitStubs.  I'll be looking into that.



----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-04-25 07:08

Message:
Logged In: YES 
user_id=80530
Originator: YES


ok, the docs and bug went in
at the same time; I just didn't
notice the doc before.

Still needs fixing.  Who's
got a round tuit?


----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2007-04-25 06:22

Message:
Logged In: YES 
user_id=80530
Originator: YES


Hey!  Documenting the brokenness
doesn't count as fixing it!


----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2006-10-17 11:45

Message:
Logged In: YES 
user_id=80530


I'm certain that making
guesses about what users
might mean is the wrong
approach.  

TIP 268 offers a well-specified
langauge for expressing exactly
what version requirements are
desired.  Users of Tcl 8.5 should
use that language to say what they
mean.  We should avoid DWIMery guessing.

For the 8.4 backport/fork, it's
possible the right solution is
for Tcl to continue [package provide]ing
version 8.4 throughout all the 8.4.*
series of releases, since that's what
some collection of scripts is assuming.
That was the right answer for Tk.
Are you making use of Tcl patchlevel
requirements in that backport?


----------------------------------------------------------------------

Comment By: Jeffrey Hobbs (hobbs)
Date: 2006-10-16 21:04

Message:
Logged In: YES 
user_id=72656

I'm not so sure that it is a bug for 8.5 either.  Perhaps it
is that -exact is just not good enough when compared to what
we have now for version handling.

If a user asks for "1.2", are they asking for "1.2.0.0..."
or are they asking for "1.2.x but not 1.3 or 1.1"?  I
believe 98% are asking for the latter.

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2006-10-16 20:59

Message:
Logged In: YES 
user_id=80530


It might make sense for
the backport/fork in the
8.4.* series (the compatibility
calculus differs there);  I'd
have to think more carefully to be
certain.

I'm confident it doesn't
make sense going forward,
so at a minimum, this code
doesn't belong in the
HEAD branch.

FWIW, though this is a bug,
I don't think it's so critical
to hold 8.5a5 for, so long as
it is otherwise ready.


----------------------------------------------------------------------

Comment By: Jeffrey Hobbs (hobbs)
Date: 2006-10-16 15:59

Message:
Logged In: YES 
user_id=72656

I talked with aku about making at least the 8.4 backport
support -exact 1.2 allowing 1.2.x IFF 1.2.0 was not
requested.  In other words, exactness should only be to the
level of specificity of the request.  Did that make sense?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=1578344&group_id=10894