tcl-core

[TCLCORE] Another path to case-tolerant package names

[<dg...@ni...>]

I'm about 30 messages buried in e-mail so forgive
me if someone else has already come forward with
the same solution.

First, make [package] as it is now into an ensemble
in the conventional way, redirecting subcommands
to the corresponding [tcl::package::*] command.
([package provide] -> [tcl::package::provide]).

Next define this proc:

proc ::tcl::package::Ifneeded {name v script} {

     # omit redirect of the "no script argument" case

     # do what we do now
     ifneeded $name $v $script

     # For all lowercase package names, we're done
     set lcname [string tolower $name]
     if {$name eq $lcname} {return}

     # package names with more "interesting" case get
     # entered into the database a second time in
     # all lowercase form, using chaining to pull in
     # the real package name:

     set lcscript [list package require $name $v-$v]
     append lcscript \n[list package provide $lcname $v]
     ifneeded $lcname $v $lcscript
}

Then change the ensemble redirection so that
[package ifneeded] routes to [::tcl::package::Ifneeded].

Now no changes need to be made to any existing package.
Any package that doesn't have an all lowercase name will
automatically make itself available in both original
and all lowercase naming variants.

Doing this with ensemble redirections means the ability
to completely restore the legacy mode is possible.  We
can ponder whether a command to set "compat mode" for that
is worthwhile, but with less urgency since even if we don't
do it we leave the tools in place at script level for
those users who must have it to make it so for themselves.

No begging and pleading and hoping for package authors
to get their act together (a proven losing strategy), and
at the same time, no destructive refusal to use the names
those authors have asked to use.  Simply the addition of
a bridge.

DGP

Re: [TCLCORE] Another path to case-tolerant package names

[Andreas K. <and...@ac...>]

> I'm about 30 messages buried in e-mail so forgive
> me if

> someone else has already come forward with
> the same solution.

No, you are the first.

> First, make [package] as it is now into an ensemble
> in the conventional way, redirecting subcommands
> to the corresponding [tcl::package::*] command.
> ([package provide] -> [tcl::package::provide]).

> Next define this proc:

> proc ::tcl::package::Ifneeded {name v script} {
>
>      # omit redirect of the "no script argument" case

>      # do what we do now
>      ifneeded $name $v $script
>
>      # For all lowercase package names, we're done
>      set lcname [string tolower $name]
>      if {$name eq $lcname} {return}

>      # package names with more "interesting" case get
>      # entered into the database a second time in
>      # all lowercase form, using chaining to pull in
>      # the real package name:

>      set lcscript [list package require $name $v-$v]
>      append lcscript \n[list package provide $lcname $v]
>      ifneeded $lcname $v $lcscript
> }

> Then change the ensemble redirection so that
> [package ifneeded] routes to [::tcl::package::Ifneeded].

> Now no changes need to be made to any existing package.
> Any package that doesn't have an all lowercase name will
> automatically make itself available in both original
> and all lowercase naming variants.

Correct.

The only packages not handled by the above are those going through the C
interface, i.e. Tcl_PkgRequire() and variants, to register themselves.

These can be handled however as well, by redirecting the
'::tcl::package::require' implementation.

If a name is not found by the original 'require' use names' to look for
camel-case variations of the  name. If there is one add the necessary bridge
and redo the 'require'. We could restrict our search to be done only if the
required name is all-lower case. I.e. using the CamelCase name asks for
strict compliance.

proc ::tcl::package::Require {name args} {

	if {![catch {
		require $name {*}$args
	} res opts]} {
		return $res
	}

	# name was camel-case, we are strict, rethrow the error
	set lcname [string tolower $name]
      if {$name ne $lcname} {return ... $res ... $opts}

	# name was all-lowercase, look for camel-case variations, take first (or
error if ambigous?)
	foreach p [names] {
		if {[string tolower $p] ne $name} continue

		# ... create and enter bridge ...

		# redo query
		return [require $name {*}$args
	}

	# nothing found, rethrow error
	return ... $res ... $opts
}


> Doing this with ensemble redirections means the ability
> to completely restore the legacy mode is possible.  We
> can ponder whether a command to set "compat mode" for that
> is worthwhile, but with less urgency since even if we don't
> do it we leave the tools in place at script level for
> those users who must have it to make it so for themselves.
>
> No begging and pleading and hoping for package authors
> to get their act together (a proven losing strategy), and
> at the same time, no destructive refusal to use the names
> those authors have asked to use.  Simply the addition of
> a bridge.


--
	Andreas Kupries <and...@Ac...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

Re: [TCLCORE] Another path to case-tolerant package names

[<dg...@ni...>]

  ...Revise [package ifneeded] so that...
>> Any package that doesn't have an all lowercase name will
>> automatically make itself available in both original
>> and all lowercase naming variants.

A few loose ends...

This scheme also requires a change in the [package unknown]
contract.  This is common to every scheme with aims similar
to TIP 339, because the handler has to conform to the new
expectation that it will seek out [package ifneeded] evaluations
from all case variants before giving up and permitting [package
require] to report "not found".  There's no case tolerance scheme
that can escape that.  (Well, none other than calling the
[package unknown] handler 2**[string length $name] times with
all the case variants!)

The other hole I can see in what I sketched is the mode where
[package require] succeeds because the requirement is met by an
already loaded package.  I *think* that omission is what you're
getting at below.  Or maybe the combination of these two matters?

Quoting "Andreas Kupries" <and...@ac...>:
> The only packages not handled by the above are those going through the C
> interface, i.e. Tcl_PkgRequire() and variants, to register themselves.

...but unless I'm missing something, this has nothing at all
to do with C interface vs. Tcl interface.

> These can be handled however as well, by redirecting the
> '::tcl::package::require' implementation.

Following your lead, try this:

proc ::tcl::package::Require {name args} {
     set lcname [string tolower $name]

     # Required name not all lower case.
     # No collision with phony entries possible.
     # Use the normal subcommand.
     if {$lcname ne $name} {
         return [require $name {*}$args]
     }

     if {[package provide $name] ne {}} {
         # Requested package is already provided.
         # Whether that's real or phony, the
         # normal subcommand can handle it right
         return [require $name {*}$args]
     }

     # Tricky case.  $name has not been provided,
     # but some case variant might have been, in
     # some way that avoided contact with the
     # [package ifneeded] wrapping trickery.  So
     # search for such a thing...
     foreach n [lsearch -all -inline -nocase [package names] $name] {
         set v [package provide $n]
         if {$v ne {}]} {
             # Fill in the missing phony entry
             package provide $name $v
             break
         }
     }

     # Any missing match is now filled in, so
     # the normal subcommand is now ok to call,
     # and permit it to probe [package ifneeded]
     # entries and invoke [package unknown] handlers.
     return [require $name {*}$args]
}

[package present] needs the same thing.

As I've sketched this, there's one troubling implication,
and that is there is one group who are measurably
worse off, and that is the authors of packages who
have chosen all lowercase names.  Up to now, they've
been in the same situation as any package author.  They
need their users to require them by exactly the
correct name, and have to worry about collision with
another author who chose precisely the same name.

After this revision, the packages with all lowercase
names now have to worry about colliding with the
whole case-insensitive set of matching package names,
a bigger potential for collision.  ("trofs" has to
worry about collision with "Trofs")  In contrast,
everyone else is likely better off, with users
now able to require them in all lowercase form,
so long as their name is unique in that broader
collection ("Itcl" gets all those users who can
only spell "itcl"), but even if there is a collision
preventing that, at least they are no worse off and
the precisely correct requires from before continue
to work.

So the incentives push package authors mildly *away*
from using all lowercase names, while encouraging
[package require] callers to make use of all lowercase
names.  I think since the number of package users should
be much larger than package providers, it makes most
sense to give the typing convenience to them.  The other
impact is to encourage all package authors to aim for
names that are globally unique even without case to
distinguish.  And that's a helpful preparation step
for conversion to Tcl Modules.

DGP

Re: [TCLCORE] Another path to case-tolerant package names

[<dg...@ni...>]

Quoting dg...@ni...:
> As I've sketched this, there's one troubling implication,
> and that is there is one group who are measurably
> worse off, and that is the authors of packages who
> have chosen all lowercase names. ...

Daniel Steffen has pointed out to me in the chat the
source of my difficulty.  By stubbornly trying to
solve this in Tcl only, I'm limited to the storage
present in the current [package] implementation,
which is a single table.  My so-called "real" and
"phony" entries are forced into one table which
causes the collisions and confusions I note above.

Drop into C and the addition of a second table isn't
hard.  It was even part of the TIP 339 Implementaion,
if I followed that right.  Doing so can keep the
existing table functioning undisturbed.

DGP

Re: [TCLCORE] Another path to case-tolerant package names

[Andreas K. <and...@ac...>]

> Quoting dg...@ni...:
> > As I've sketched this, there's one troubling implication,
> > and that is there is one group who are measurably
> > worse off, and that is the authors of packages who
> > have chosen all lowercase names. ...
>
> Daniel Steffen has pointed out to me in the chat the
> source of my difficulty.  By stubbornly trying to
> solve this in Tcl only, I'm limited to the storage
> present in the current [package] implementation,
> which is a single table.  My so-called "real" and
> "phony" entries are forced into one table which
> causes the collisions and confusions I note above.
>
> Drop into C and the addition of a second table isn't
> hard.  It was even part of the TIP 339 Implementaion,
> if I followed that right.  Doing so can keep the
> existing table functioning undisturbed.

Yes. The reference implementation used a second hashtable.

The original table kept all the exact package names as keys.

The second table contained the case-folded form as keys.

Both tables shared the 'struct Package' however.

All packages with the same case-folded form are linked together, the first
in the list is the one used where the TIP specified 'first registered' ...


Note that using a pure-Tcl solution _does not_ prevent us from using a
second table. We can have one, it would just be an array variable.

--
	Andreas Kupries <and...@Ac...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

Re: [TCLCORE] Another path to case-tolerant package names

[Jan N. <jan...@gm...>]

2008/12/12  <dg...@ni...>:
>     # Tricky case.  $name has not been provided,
>     # but some case variant might have been, in
>     # some way that avoided contact with the
>     # [package ifneeded] wrapping trickery.  So
>     # search for such a thing...

This should not be possible: Doing this in C means
that there is no way to circumvent it: Any package
providing "FooBar" should immediately provide
"foobar" as phony entry as well.

Such kind of search is asking for trouble :-/

> Drop into C and the addition of a second table isn't
> hard.  It was even part of the TIP 339 Implementaion,
Maybe even simpler than a second table is just
a flag in each entry, indicating whether the entry
is phony or not, or better an entry indicating the
'companion entry'. Then we can, in case of an
'unload' always remove both the real and the
phony entry.

Bringing an 'unload' into account as well, that
might make things more tricky.......

Regards,
         Jan Nijtmans

Regards,
          Jan Nijtmans

Re: [TCLCORE] Another path to case-tolerant package names

[Daniel A. S. <da...@us...>]

On Fri, Dec 12, 2008 at 09:45, Jan Nijtmans <jan...@gm...> wrote:

> Maybe even simpler than a second table is just
> a flag in each entry, indicating whether the entry
> is phony or not, or better an entry indicating the
> 'companion entry'.

No, if you want to stay backwards compatible, there is no way around a
second hash table IMO, otherwise you cannot avoid key collision of
ordinary entries (e.g. key "tk" for package "tk") with phony entries
(e.g. case-folded key "tk" for package "Tk")

Concretely, the idea discussed with Don on the chat this morning was
the following:

- in Tcl_PkgProvide() add an entry keyed by the case-folded name of
the provided package to a new fallback table (in addition to the
existing insertion of the package into iPtr->packageTable).
- amend PkgRequireCore() to check this fallback table with the
case-folded name iff the ordinary package lookup has failed (including
the [package unknown] invocation & subsequent recheck).
- amend the [package unknown] handler contract (and the existing core
handlers) to include a search for any case-variant of the asked-for
package name iff the search for the exact asked-for name has failed.
- after all calls to Tcl_FindHashEntry(&iPtr->packageTable,...) in
tclPkg.c (except for PKG_NAMES),  lookup the case-folded name in the
fallback table iff the packageTable lookup has failed.

On a install with no case conflicts in package names, this would
result in a [package require] that is completely case-insensitive
w.r.t package names.
With packages present whose name only differs by case however,
[package require]s of their exact names would continue to get the
right package, i.e. have the same result as it does currently (and
[package require] of non-existent case variants would get the last
[package provide]d one).

This scheme would seem to provide most of the benefits of TIP339 while
being fully backwards compatible w.r.t existing successful [package
require]s in the presence of case conflicting package names.

Of course existing _failing_ [package require]s might become
successful under this, which could be an issue for some code, but that
should still be a much lesser incompatibility than existing successful
[package require]s possibly loading a different package than
intended...

Cheers,

Daniel

Re: [TCLCORE] Another path to case-tolerant package names

[Jan N. <jan...@gm...>]

2008/12/12 Daniel A. Steffen <da...@us...>:
> No, if you want to stay backwards compatible, there is no way around a
> second hash table IMO, otherwise you cannot avoid key collision of
> ordinary entries (e.g. key "tk" for package "tk") with phony entries
> (e.g. case-folded key "tk" for package "Tk")

Yes, you are right. And performance is no issue here, because
the fallback search is only done if the first one failed.

> This scheme would seem to provide most of the benefits of TIP339 while
> being fully backwards compatible w.r.t existing successful [package
> require]s in the presence of case conflicting package names.

Right!

> Of course existing _failing_ [package require]s might become
> successful under this, which could be an issue for some code, but that
> should still be a much lesser incompatibility than existing successful
> [package require]s possibly loading a different package than
> intended...

I Guess I'm getting enthousiastic about this idea. The only thing
that needs to be verified if Tcl modules still work: Will
a "package require foobar" find the file FooBar-1.0.0.tm?

Regards,
        Jan Nijtmans

Re: [TCLCORE] Another path to case-tolerant package names

[Jan N. <nij...@us...>]

2008/12/12 Jan Nijtmans <jan...@gm...>:
> 2008/12/12 Daniel A. Steffen <da...@us...>:
>> No, if you want to stay backwards compatible, there is no way around a
>> second hash table IMO, otherwise you cannot avoid key collision of
>> ordinary entries (e.g. key "tk" for package "tk") with phony entries
>> (e.g. case-folded key "tk" for package "Tk")

I am stubborn enough to experiment with a solution that doesn't need
a second hash table. The collision is prevented by removing the phony
entries as soon as a real entry for the same package name is
encountered. Apart from that, I tried to be as close as possible
to the ideas discussed in this thread.

The resulting patch is available at:
https://sourceforge.net/tracker2/?func=detail&aid=2432057&group_id=10894&atid=310894
but I'll outline the idea here.

The idea is, not trying to make package handling case-insensitive, but
in stead only handle the situation that someone provides a package
"FooBar" and we want to make it available under the name "foobar"
as well. So, "foobar" is a phony package. We do that by changing
"package require FooBar" such that it automatically provides "foobar"
as wel, if it was not provided before. In addition, providing a new
real "foobar" package, where a phony "foobar" was already provided,
no longer is an error, it just ignores the phony package.

The "ifneeded" is changed to work as if
    package ifneeded FooBar $version somescript
automatically does:
    package ifneeded foobar $version [list package require -exact
FooBar $version]
as well. The extra "package provide" (as in Don's original idea) is not
neccesary, because that is handled by the "package provide" which
is already in the loaded package. The nice thing here is that from the
ifneeded script we can immediately see if the package is phony or
not. For phony packages the ifneeded script is always a 5-element
list in which the 4th element is the name of the real package!
That's transparent!

The implementation starts with extending the Package structure with a flag
"real", and extend the FindPackage with an extra "Package **phony" argument.
Every time an entry is created in the hashtable for "FooBar", another
entry "foobar"
is created as well. If the package name is lowercase, FindPackage functions
as before, otherwise the additional phony package is returned as well as
the real package. This function is used in a lot of other places.

Then the changes to "package ifneeded" and "package provide" are rather
streight-forward. The only possible incompatibility is that extra phony
package names can magically apprear and disappear: every time a
real package name might conflict with a phony package, the situation
is handled as if the phony package simply doesn't exist.

The only problem there is left, is that autoMkindex.test crashes, I'm
still trying to debug that (if someone has a good debugger, I appreciate
help in tracing this down, somewhere still is a bug). All other
tests pass, even some additional ones that I wrote (but didn't
include in this patch, I should have done that......).

It's late now, so my apologies if this mail is a little condensed....

Regards,
     Jan Nijtmans

Re: [TCLCORE] Another path to case-tolerant package names

[Jan N. <nij...@us...>]

2008/12/16 Jan Nijtmans <nij...@us...>:
> The only problem there is left, is that autoMkindex.test crashes, I'm
> still trying to debug that (if someone has a good debugger, I appreciate
> help in tracing this down, somewhere still is a bug).

Found the bug: forgot to check pkgPtr->version != NULL before
trying to ckfree() it....

Uploaded a new patch to:
https://sourceforge.net/tracker2/?func=detail&aid=2432057&group_id=10894&atid=310894
which passes all tests, including a few added ones.

Any feedback is welcome. Is this a valid alternative to TIP #339?

Regards,
      Jan Nijtmans

Re: [TCLCORE] Another path to case-tolerant package names

[Donald G P. <dg...@ni...>]

Jan Nijtmans wrote:
> I am stubborn enough to experiment with a solution that doesn't need
> a second hash table. 

I think I spoke unclearly before.  The real constraint I had
imposed on myself was to achieve what was needed completely at
script level.  Essentially prove that the existing [package]
command was flexible enough (with minimal recasting in ensemble
form) to provide the desired function.  That limitation implies
sticking to one table, but that's not all.

Certainly dropping into C and changing the structures held in
the table, one can achieve what's needed without a second hash table.

> The resulting patch is available at:
> https://sourceforge.net/tracker2/?func=detail&aid=2432057&group_id=10894&atid=310894
> but I'll outline the idea here.

The patch crashes in the test suite.  Not commit ready.

> The idea is, not trying to make package handling case-insensitive, but
> in stead only handle the situation that someone provides a package
> "FooBar" and we want to make it available under the name "foobar"
> as well. 

Ok, so fundamentally the approach is leave [package] unchanged in its
most internal details -- keep it as a fundamentally case sensitive
system -- but add on just enough more to provide illusion of
case insensitivity as a crutch for folks more comfortable with that.
If it worked, I think I could accept that.

However, it doesn't currently work, so I'll mention that I think the
opposite aim would be better.  The momentum as I perceive it is that
package names should have been case insensitive, and packages should
have always been provided all lowercase, and we'd be best to move to a
scheme that in its fundamentals is a case insensitive system, with the
addition of just enough more so that we break as little existing
code as we can.  Then, if the aim is for [package] in, say Tcl 9,
to really become completely case insensitive, we've built a better
bridge to that endpoint.

I have some ideas how to do that, but I don't have a patch to offer,
and I don't know when I can promise time to produce one.  So, if
patch 2432057 can get corrected quickly, I'll defer to that.

-- 
| Don Porter          Mathematical and Computational Sciences Division |
| don...@ni...             Information Technology Laboratory |
| http://math.nist.gov/~DPorter/                                  NIST |
|______________________________________________________________________|

Re: [TCLCORE] Another path to case-tolerant package names

[Andreas K. <and...@ac...>]

> On Fri, Dec 12, 2008 at 09:45, Jan Nijtmans
> <jan...@gm...> wrote:
>
> > Maybe even simpler than a second table is just
> > a flag in each entry, indicating whether the entry
> > is phony or not, or better an entry indicating the
> > 'companion entry'.

> No, if you want to stay backwards compatible, there is no way around a
> second hash table IMO, otherwise you cannot avoid key collision of
> ordinary entries (e.g. key "tk" for package "tk") with phony entries
> (e.g. case-folded key "tk" for package "Tk")

> Concretely, the idea discussed with Don on the chat this morning was
> the following:

Daniel, Don, can you both please have a look at the TIP 339 reference
implementation (339-ri) and check how much of the code implements the
behaviour you are specifying below, and for the parts which don't how
difficult to a change would be ?


> - in Tcl_PkgProvide() add an entry keyed by the case-folded name of
> the provided package to a new fallback table (in addition to the
> existing insertion of the package into iPtr->packageTable).

Yes, that is done in 339-ri

> - amend PkgRequireCore() to check this fallback table with the
> case-folded name iff the ordinary package lookup has failed (including
> the [package unknown] invocation & subsequent recheck).

Ok, this is not done that way in 339-ri IIRC. That is where the -strict
option came in selecting the match behaviour.

This part would have to change per your spec.


> - amend the [package unknown] handler contract (and the existing core
> handlers) to include a search for any case-variant of the asked-for
> package name iff the search for the exact asked-for name has failed.

This should be in the 339-ri as well ... Actually, not quite. The only
handler which needed a change was for Tcl Modules, and it now considers the
existence of 'Foo' and 'fOO' files as problem, claiming package not found.
Changing that should be not too difficult. Look at the 'FileExists' code, it
has to be expanded to simply track all possible matches instead of aborting
when multiple appear.

The handler for regular packages needed no change because it always does a
full scan of all pkgIndex.tcl files, thus pulling  in all case-variants of
whatever automatically as part of its search for everything.


> - after all calls to Tcl_FindHashEntry(&iPtr->packageTable,...) in
> tclPkg.c (except for PKG_NAMES),  lookup the case-folded name in the
> fallback table iff the packageTable lookup has failed.

I believe that needs changing in 3390ri as well.


> On a install with no case conflicts in package names, this would
> result in a [package require] that is completely case-insensitive
> w.r.t package names.

good.

> With packages present whose name only differs by case however,
> [package require]s of their exact names would continue to get the
> right package, i.e. have the same result as it does currently (and
> [package require] of non-existent case variants would get the last
> [package provide]d one).

sounds a bit like dwimmery, however also as explainable, therefore
acceptable to me.

> This scheme would seem to provide most of the benefits of TIP339 while
> being fully backwards compatible w.r.t existing successful [package
> require]s in the presence of case conflicting package names.

> Of course existing _failing_ [package require]s might become
> successful under this, which could be an issue for some code, but that
> should still be a much lesser incompatibility than existing successful
> [package require]s possibly loading a different package than
> intended...

sounds good.

Again, are you willing to look over the 339-ri and see what would need
modification ?

All the stuff for -strict would have to be removed, but the data structures
seem to be (partially) salvage-able.

--
	Andreas Kupries <and...@Ac...>
	Developer @ http://www.ActiveState.com
	Tel: +1 778-786-1122

Re: [TCLCORE] Another path to case-tolerant package names

[Jan N. <jan...@gm...>]

2008/12/11  <dg...@ni...>:
> I'm about 30 messages buried in e-mail so forgive
> me if someone else has already come forward with
> the same solution.
No, I didn't see this solution before.

Sounds good, only one remark:

- What if there are two packages e.g. "tcltest" and "tcltest", where
  the lower-case package is found first. Then what happens is:
         package ifneeded tcltest script1
         package ifneeded Tcltest script2
         package ifneeded tcltest script2\npackage provide tcltest ....
  So, the lower-case "package ifneeded" for Tcltest overwrites the
  original "package ifneeded" for "tcltest", so "tcltest" can never be
  loaded any more.
  Solution: before doing the second "package ifneeded", check if
  there already is one. If so, don't overwrite the previous.
The nice thing is that this works for C packages and for Modules
as well: the script that does the "package ifneeded" determines
the casing of the Module filename, nothing changes there.
That's very nice.

I don't think that using an ensemble for this is even neccessary.
This is worth investigating further.

Regards,
        Jan Nijtmans

Re: [TCLCORE] Another path to case-tolerant package names

[<dg...@ni...>]

Quoting "Jan Nijtmans" <jan...@gm...>:
> - What if ... [there is a "real" package with an
     all lowercase name that gets its slots in the
     package database polluted by the entries for
     the phony package added just for this caseless
     finding trickery?]

>   Solution: before doing the second "package ifneeded", check if
>   there already is one. If so, don't overwrite the previous.

That's pointed in the right direction.  I believe the true
solution is not quite that simple, but certainly doable.
(You want the entries for real packages to win over the phony
entries, and "first claim wins" won't accomplish that.)

> I don't think that using an ensemble for this is even neccessary.

No, not necessary.  We could just modify the C implementation
of [package ifneeded] to the new spec directly.

The advantage of the ensemble approach is that it "exposes
the wires" to script level, so then things are more easily
switchable back to the old way, or to other ways.  A useful
capability when we're rushing a design and there's at least
some reason to doubt we've settled on the unquestionably
best solution.

DGP