[Tcl-bugs] [ tcl-Bugs-803489 ] Tcl_FindNamespace problem in the Stubs table

[SourceForge.net <no...@so...>]

Bugs item #803489, was opened at 2003-09-09 19:10
Message generated for change (Settings changed) made by dgp
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=803489&group_id=10894

Category: 21. [namespace]
Group: develop: 8.5a0
Status: Open
Resolution: Invalid
>Priority: 7
Submitted By: David Gravereaux (davygrvy)
Assigned to: Don Porter (dgp)
Summary: Tcl_FindNamespace problem in the Stubs table

Initial Comment:
Tcl_FindNamespace now has a new entry in the public 
table for 8.5.  The old one still exists in the private 
table, too, but isn't visible due to .  The public tclDecls.h 
is included first.

All fine so far, but any extension that links with the new 
Stubs table blows chucks when loaded into an older 
core.  Why?  tclStubsPtr->tcl_FindNamespace is NULL.  
I'd rather have it still be tclIntStubsPtr-
>tcl_FindNamespace, still.

Solution?  Not sure..

tclDecls.h:
#ifndef Tcl_FindNamespace
#define Tcl_FindNamespace 	(tclStubsPtr->tcl_FindNamespace) /* 514 */
#endif

tclIntDecls.h:
#ifndef Tcl_FindNamespace
#define Tcl_FindNamespace 	(tclIntStubsPtr->tcl_FindNamespace) /* 117 */
#endif


----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2003-09-13 18:12

Message:
Logged In: YES 
user_id=79902

Eh?  We're not backporting the TIP (or, for that matter,
changes to the stub architecture) into 8.4 since that would
just make sure that people linking against a release would
have problems (as opposed to people playing around with
pre-alpha stuff!) and we want new code to use the new stub
positions. While I am contemplating adding a #define that
will allow the API user to force the use of the old API,
that's not done yet.

You were using a private API, now you are going to be using
a public API (which has coincidentally the same names due to
a cock up in the private API naming.) There is no reasonable
reason to assume that anything compiled against the new
public version will work against the old private one.  I
might make it work in the future, but IMHO that's an
unsupported bonus.

Won't fix tonight though.  Too tired ;)


----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-13 17:49

Message:
Logged In: YES 
user_id=7549

The technical difficulty can be described in an analogy.

The door remains identical (same sig), but the locks have 
been changed (new slot in public).  The old key still works in 
the new locks, but the new key won't work in the old locks 
(new slot is NULL, old slot hidden from view).

If tclStubLib.c could fill in the new slot when NULL, both keys 
will work in both locks.  As the memory for where the Stubs 
table is located is in static memory owned by the core, a new 
table of the type the extension understands will need to be 
made that it copies into and fills from the old into the new 
and forwarding the new public ones from the the private table 
when required.  Ex:

memcpy(newTclStubsPtr, tclStubsPtr, sizeof(TclStubs));
if (newTclStubsPtr->tcl_FindNamespace == NULL) {
    newTclStubsPtr->tcl_FindNamespace = tclIntStubsPtr-
>tcl_FindNamespace;
}

The difficulty I see is not knowing the size of the table. A 
tclStubsPtr->size does not exist and is assumed larger than it 
is.  Now if magic got replaced to a sizeof(TclStubs) by 
tclStubInit.c that would help, but..   Complicated, but 
probably not impossible to work around.

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2003-09-11 13:05

Message:
Logged In: YES 
user_id=80530


Well, I understand why it's desirable for
an extension author to have both things:
1) Use of the latest Tcl for himself; 2) Support
for the earliest Tcl possible used by the users
of his compiled, stub-enabled extensions.

It's a bit less clear to me, given your talk of
changing struct sizes whether support is
100% feasible.  Even less so if we're talking
about extensions that intrude into the Tcl internals.
(for example, http://mini.net/tcl/3708 )

Another thing that's clear to me is that we are
suffering from bad decisions made long ago,
by none of us, so we could benefit from a lot
of mutual patience looking into this.

There's an extra complexity here that, technically,
until TIP 139, Tcl_FindNamespace has been a
private interface, and we've been fairly ruthless
about dropping compatibility of private interfaces
at minor releases.  So the current HEAD is
justifiable.  From that perspecitve, extensions
that call the newly public Tcl_FindNamespace should
just be sure to Tcl_InitStubs(interp, "8.5", 0); because
Tcl_FindNamespace first appears in Tcl 8.5.

That said, Tcl_FindNamespace and friends are
used by many good and faithful Tcl'ers, and Tcl
extensions, and it's worthwhile to find a better way
to accomodate them if we can.

Technically, this seems similar to the issues
in TIP 27 (CONSTification).  A symbol is 
defined in two consecutive releases of Tcl to
mean different things, and we have compatibility
problems with people who did not expect it to
change.  Seems we could adopt a compatibility support
simlar to the TIP 27 compatibility support. with a
USE_PRIVATE_NAMESPACE_API
macro, that when #define 'd, would suppress the
definition of the newly exposed and not re-named routines
by tcl.h, so that they'd be available for definition by
tclInt.h for those naughy extensions that are #include-ing it.

The default would be to not #define
USE_PRIVATE_NAMESPACE_API, and the
new Namespace routines would have their
new, and future definitions, of pointing into the
public stub table.  Default operation ought to
be the recommend usage going forward, not
looking back.



----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2003-09-11 04:09

Message:
Logged In: YES 
user_id=79902

I refuse to fix it because I believe you are trying to do
something that is not supported.  If you build against 8.x
(x=[1..4]) then you can load into 8.5 and that will work. 
But building against one version and loading into a previous
version?  That's peverse.  It won't work in general.  This
is just yet another reason why not.  (And if you use any of
the other namespace APIs that I didn't expose, if I get
around to exposing those they may well be completely
different anyway.)

dgp: Why on *earth* would we want to support that form of
compatability?  Give me a good enough reason and I'll find a
way around this, but right now I think it is a feature we do
*not* want.  (davygrvy: this is a policy decision so please
do not change anything until we have had some more
clarification, and even then I volunteer to do it since I've
grokked the stubs generator fairly recently.)

----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-10 22:56

Message:
Logged In: YES 
user_id=7549

Sorry, this is a big bug.  So I now have to source 
tclIntDecls.h prior to to tcl.h to solve this?

My app doesn't touch channel types.  The signature of 
Tcl_FindNamespace hasn't changed, so please don't change 
the slot.  Congrats, you have succesfully broken something 
that remains identical.  True, the old is still there, but hidden 
from me now due to the order of the include files.

Yet more stupid macros will be needed to support the 
backward compatibility.. Oh the joy..

If you refuse to fix it, I'll do it then.

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2003-09-10 08:57

Message:
Logged In: YES 
user_id=79902

The function is still in the old slot and there has never
been a guarantee that internal stubs will be perfectly
maintained.  In 8.4 and before, Tcl_FindNamespace is an
internal function; anyone #including tclInt.h does not have
as strong a guarantee of compatability.

Note that since there have been structure size alterations
(e.g. in channel types), it is not really generally safe to
load a stubbed library into an earlier version of Tcl, no
matter how you call Tcl_InitStubs.

IMHO, this is pilot error, *not* a bug.

----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-10 08:30

Message:
Logged In: YES 
user_id=7549

Stubs are supposed to gain, never lose slots.  You moved my 
slot.  I know this code works in 8.1, therefore, I state it:

#ifdef USE_TCL_STUBS
    if (Tcl_InitStubs(interp, "8.1", 0) == 0L) {
	return TCL_ERROR;
    }
#endif

This change breaks my stuff.  Please don't make a needed 
private->public move break existing code.  I'm all for the API 
shift to the public, but could you leave the old slot position 
the same?

----------------------------------------------------------------------

Comment By: Donal K. Fellows (dkf)
Date: 2003-09-10 08:03

Message:
Logged In: YES 
user_id=79902

Could someone explain why people might have an expectation 
that an extension compiled with 8.x might work with 8.y when 
x>y?  I thought that was a compatability case which we were 
explicitly not supporting...

----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-10 00:40

Message:
Logged In: YES 
user_id=7549

Just before I reverted, I found a quick fix:

#include "tclInt.h"

#ifdef Tcl_FindNamespace
#undef Tcl_FindNamespace
#define Tcl_FindNamespace (tclIntStubsPtr->tcl_FindNamespace)
#endif
.
.
.

That'll work for me for now..

----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-10 00:12

Message:
Logged In: YES 
user_id=7549

until this gets fixed, I'm dropping back to 8.4 for extension 
builds.  It's blowing up everything I've got in progress.

----------------------------------------------------------------------

Comment By: David Gravereaux (davygrvy)
Date: 2003-09-09 19:20

Message:
Logged In: YES 
user_id=7549

How about a new command added to genStubs.tcl:

forward generic Int {
    Tcl_Namespace *Tcl_FindNamespace(Tcl_Interp *interp, 
CONST char *name,
	    Tcl_Namespace *contextNsPtr, int flags)
}

and doesn't write write a new slot to the table, and defines 
the name to be:

#ifndef Tcl_FindNamespace
#define Tcl_FindNamespace 	(tclIntStubsPtr->tcl_FindNamespace) /* 514 */
#endif

even in the public tclStubs.h.  Would that be cart before the 
horse?

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2003-09-09 19:18

Message:
Logged In: YES 
user_id=80530


Donal, your genstubs trickery handled
forward compatibility fine, but as
davygrvy points out, it leaves out support
for backward compatibility.  (Compile an
extension against 8.5 headers, but [load]
it into an 8.4 interp).  

I know the wrapper solution is tedious,
but it appears to be the way to go.

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2003-09-09 19:15

Message:
Logged In: YES 
user_id=80530


Ah.  Never mind.  I get the trouble now.


----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2003-09-09 19:12

Message:
Logged In: YES 
user_id=80530


Have you re-run 'make genstubs'
since your last update?

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=110894&aid=803489&group_id=10894