#3720 auto_path changes and avail package cacheing issues

obsolete: 8.5a6
Don Porter

Package managing code might in certain cases not notice the availability of a more recent version of a package X that was made available by modifying auto_path at runtime if package manager have already seen any version of package X at the scanning stage.

How to trigger:
1) We have a package named, say, "foo" somewhere accessible via $auto_path;
2) Then by some means (usually by calling [package require something]) we cause the package manager to make its run through the $auto_path scanning the packages. After that the package manager will know that the package "foo" is available.
3) Then we add another direcroty to auto_path which makes some more recent version of the same package "foo" available to the package manager. (Use case: we want to locally override a certain package in the standard package hierarchy).
4) Then [package require foo] will fetch the original version (not the latest) since the package manager won't rescan the directories listed on auto_path.

From the Tclers' chatroom:
[00:14]<jenglish> kostix - the default [package unknown] handler does keep track of auto_path adjustments -- if any pkgIndex.tcl diddles with auto_path (which, really, they shouldn't, but some do anyway...) it will scan the new directories too.

[00:18]<jenglish> ... but IIRC the default [package unknown] handler has some short-circuiting logic; it might stop scanning for pkgIndex.tcl files once it finds a package that satisfies the requirements.

The ways to overcome the problem are:
1) Add [package forget foo] before calling [package require foo];
2) Specify the exact version in [package require] to make [package unknown] be not happy with the version of "foo" it already known about.

My poposition is to make the package manager rescan the hierarchy on any change of auto_path (may be it's worth remembering the sorted list of normalized paths from auto_path and compare it with the similar list aquired from the changed auto_path to track changes which don't really change things). Changes to auto_path are rare and probably such change in behaviour won't lead to slowdowns.

This issue is also confirmed to exist in 8.4.14 and .15


  • a minimal testcase

  • Logged In: YES
    Originator: YES

    Correction: I don't mean "rescan packages when auto_path changes", I mean "Invalidate the cache of known packages when auto_path changes" so that any subsequent [package require] (or whatever) make the package manager to rescan the list of packages.

  • Don Porter
    Don Porter

    Logged In: YES
    Originator: NO

    since I'm short on time, here's what
    I had to say on this in the chat:

    dgp re 1733117 my intent is to follow up in detail in the Tracker...
    [12:02] dgp but briefly the answer is "not a bug"

    [12:03] kbk Hmm, "not a bug" perhaps but at least surprising.

    [12:04] dgp probably the most surprising thing...
    [12:04] dgp ...is that [package require $p $requirement] does *not* make any guarantee to provide the latest version available
    [12:05] dgp it only promises to satisfy the requirements passed in as arguments

    [12:05] dgp If some documentation somewhere says otherwise, then *that* is a bug
    [12:06] kbk Hmmm. OK so far. Would an explicit requirement of the version that was on the new auto_path have found it?
    [12:06] dgp so, if an app really, really, really won't be satisfies with anything other than the very latest available, they'll have to use the lower level [package] subcommands to accomplish that for themselves
    [12:06] dgp I think all the tools are there.
    [12:06] kbk (That militates in favour of aku's practice of advancing patch levels in tcllib modules with every bugfix)
    [12:08] dgp if the modified auto_path makes findable a new version of a package...
    [12:08] dgp that was not findable before...
    [12:08] dgp ...then a requirement that is only satisfied by that new version, and was not satisfied by the older ones available
    [12:08] dgp will find the new one, yes.

    [12:10] kbk ok, I'm almost satisfied... although "latest available version that satisfies requirements" would make a lot of sense.

    [12:10] kbk Otherwise, consider an app that does [package require foo 1.0; package require bar 1.0]

    [12:11] kbk where (properly unknown to the app writer), foo does [package require grill 1.1] and bar does [package require grill 1.2].
    [12:11] dgp Without thinking it fully through, I think a custom [package unknown] handler can do this

    [12:17] dgp so quickly, how a cusom [package unknown] can solve the posed problem....
    [12:17] dgp start with [::tclPkgUnknown] which searches according to the value of ::auto_path...
    [12:18] dgp augment it with a variable trace on ::auto_path that calls [package forget] when ::auto_path is written to.
    [12:18] dgp then register this modified package finder with [package unknown]
    [12:18] dgp and voila, your app has the feature requested.

    jenglish -- I think all you need is a way to say "scan this directory for available packages _now_".

    [12:21] jenglish ([lappend auto_path ...] just says "scan this directory for packages the next time somebody asks for a package you don't already know about", which is error-prone and order-dependent)

    [12:22] dgp [lappend auto_path ...; eval [package unknown] Tcl [package provide Tcl]]

    [12:22] dgp That's not 100%, but it works with [tclPkgUnknown]
    [12:23] dgp can't be 100% because a cusom [package unknown] handler is free to provide anything that satisfies requirements even if that means ignoring later available versions.

    [12:24] jenglish wait, I thought the package [package unknown] handler was only responsible for providing ifneeded scripts - not for providing the packages themselves.

    [12:25] dgp [package unknown] is for finding packages.
    [12:25] dgp when [package require] determines it can't satisfy requirement it knows about.

    [12:25] dgp The default [package unknown], [tclPkgUnknown] goes searching for every version of every package it can find.
    [12:26] dgp ...but other custom ones don't have to do that.

    [12:26] dgp They get passed the package and requirements being sought.
    [12:26] jenglish So the contract for [package unknown] handlers is: either provide the package, or register an ifneeded script that will provide the package?
    [12:26] dgp and are free to search less broadly with the aim to satisfy the stated requirement.
    [12:26] dgp For example, the Tcl Module [package unknown] routine
    [12:27] dgp no [package unknown] never [package provide]s a package
    [12:27] dgp (assuming proper, nonabusive, usage)

    jenglish TFM says: "For example, if [the package-unknown handler] is 'foo bar' [...] then Tcl will execute the command 'foo bar test 2.4' *to load the package*" (emphasis mine)

    [12:32] dgp that would probably be better rephrased.
    [12:32] dgp the task for [foo bar test 2.4] to accomplish is to run at least one [package ifneeded foo $x] command
    [12:32] dgp where [package vsatisfies 2.4 $x] is true
    [12:33] jenglish Related: calling [package forget] when auto_path changes might not be a good idea -- you don't want to forget previous [package provide]s, just the [package ifneeded]s.
    [12:33] dgp or to run no [package ifneeded] command at all, in the cases where no satisfaction can be had.
    [12:33] dgp you would pass the package name to [package forget]
    [12:34] dgp there are details to get right, you are correct.
    [12:34] dgp I think all the tools are available to accomplish that.
    [12:35] dgp when I get back to full time, perhaps I can offer a complet example

    Leaving this open since there's doc improvement likely to be useful.