Package managing code might in certain cases not notice the availability of a more recent version of a package X that was made available by modifying auto_path at runtime if package manager have already seen any version of package X at the scanning stage.
How to trigger:
1) We have a package named, say, "foo" somewhere accessible via $auto_path;
2) Then by some means (usually by calling [package require something]) we cause the package manager to make its run through the $auto_path scanning the packages. After that the package manager will know that the package "foo" is available.
3) Then we add another direcroty to auto_path which makes some more recent version of the same package "foo" available to the package manager. (Use case: we want to locally override a certain package in the standard package hierarchy).
4) Then [package require foo] will fetch the original version (not the latest) since the package manager won't rescan the directories listed on auto_path.
From the Tclers' chatroom:
[00:14]<jenglish> kostix - the default [package unknown] handler does keep track of auto_path adjustments -- if any pkgIndex.tcl diddles with auto_path (which, really, they shouldn't, but some do anyway...) it will scan the new directories too.
[00:18]<jenglish> ... but IIRC the default [package unknown] handler has some short-circuiting logic; it might stop scanning for pkgIndex.tcl files once it finds a package that satisfies the requirements.
The ways to overcome the problem are:
1) Add [package forget foo] before calling [package require foo];
2) Specify the exact version in [package require] to make [package unknown] be not happy with the version of "foo" it already known about.
My poposition is to make the package manager rescan the hierarchy on any change of auto_path (may be it's worth remembering the sorted list of normalized paths from auto_path and compare it with the similar list aquired from the changed auto_path to track changes which don't really change things). Changes to auto_path are rare and probably such change in behaviour won't lead to slowdowns.
P.S.
This issue is also confirmed to exist in 8.4.14 and .15
a minimal testcase
Logged In: YES
user_id=1350198
Originator: YES
Correction: I don't mean "rescan packages when auto_path changes", I mean "Invalidate the cache of known packages when auto_path changes" so that any subsequent [package require] (or whatever) make the package manager to rescan the list of packages.
Logged In: YES
user_id=80530
Originator: NO
since I'm short on time, here's what
I had to say on this in the chat:
dgp re 1733117 my intent is to follow up in detail in the Tracker...
[12:02] dgp but briefly the answer is "not a bug"
[12:03] kbk Hmm, "not a bug" perhaps but at least surprising.
[12:04] dgp probably the most surprising thing...
[12:04] dgp ...is that [package require $p $requirement] does *not* make any guarantee to provide the latest version available
[12:05] dgp it only promises to satisfy the requirements passed in as arguments
[12:05] dgp If some documentation somewhere says otherwise, then *that* is a bug
[12:06] kbk Hmmm. OK so far. Would an explicit requirement of the version that was on the new auto_path have found it?
[12:06] dgp so, if an app really, really, really won't be satisfies with anything other than the very latest available, they'll have to use the lower level [package] subcommands to accomplish that for themselves
[12:06] dgp I think all the tools are there.
[12:06] kbk (That militates in favour of aku's practice of advancing patch levels in tcllib modules with every bugfix)
[12:08] dgp if the modified auto_path makes findable a new version of a package...
[12:08] dgp that was not findable before...
[12:08] dgp ...then a requirement that is only satisfied by that new version, and was not satisfied by the older ones available
[12:08] dgp will find the new one, yes.
[12:10] kbk ok, I'm almost satisfied... although "latest available version that satisfies requirements" would make a lot of sense.
[12:10] kbk Otherwise, consider an app that does [package require foo 1.0; package require bar 1.0]
[12:11] kbk where (properly unknown to the app writer), foo does [package require grill 1.1] and bar does [package require grill 1.2].
[12:11] dgp Without thinking it fully through, I think a custom [package unknown] handler can do this
[12:17] dgp so quickly, how a cusom [package unknown] can solve the posed problem....
[12:17] dgp start with [::tclPkgUnknown] which searches according to the value of ::auto_path...
[12:18] dgp augment it with a variable trace on ::auto_path that calls [package forget] when ::auto_path is written to.
[12:18] dgp then register this modified package finder with [package unknown]
[12:18] dgp and voila, your app has the feature requested.
jenglish -- I think all you need is a way to say "scan this directory for available packages _now_".
[12:21] jenglish ([lappend auto_path ...] just says "scan this directory for packages the next time somebody asks for a package you don't already know about", which is error-prone and order-dependent)
[12:22] dgp [lappend auto_path ...; eval [package unknown] Tcl [package provide Tcl]]
[12:22] dgp That's not 100%, but it works with [tclPkgUnknown]
[12:23] dgp can't be 100% because a cusom [package unknown] handler is free to provide anything that satisfies requirements even if that means ignoring later available versions.
[12:24] jenglish wait, I thought the package [package unknown] handler was only responsible for providing ifneeded scripts - not for providing the packages themselves.
[12:25] dgp [package unknown] is for finding packages.
[12:25] dgp when [package require] determines it can't satisfy requirement it knows about.
[12:25] dgp The default [package unknown], [tclPkgUnknown] goes searching for every version of every package it can find.
[12:26] dgp ...but other custom ones don't have to do that.
[12:26] dgp They get passed the package and requirements being sought.
[12:26] jenglish So the contract for [package unknown] handlers is: either provide the package, or register an ifneeded script that will provide the package?
[12:26] dgp and are free to search less broadly with the aim to satisfy the stated requirement.
[12:26] dgp For example, the Tcl Module [package unknown] routine
[12:27] dgp no [package unknown] never [package provide]s a package
[12:27] dgp (assuming proper, nonabusive, usage)
jenglish TFM says: "For example, if [the package-unknown handler] is 'foo bar' [...] then Tcl will execute the command 'foo bar test 2.4' *to load the package*" (emphasis mine)
[12:32] dgp that would probably be better rephrased.
[12:32] dgp the task for [foo bar test 2.4] to accomplish is to run at least one [package ifneeded foo $x] command
[12:32] dgp where [package vsatisfies 2.4 $x] is true
[12:33] jenglish Related: calling [package forget] when auto_path changes might not be a good idea -- you don't want to forget previous [package provide]s, just the [package ifneeded]s.
[12:33] dgp or to run no [package ifneeded] command at all, in the cases where no satisfaction can be had.
[12:33] dgp you would pass the package name to [package forget]
[12:34] dgp there are details to get right, you are correct.
[12:34] dgp I think all the tools are available to accomplish that.
[12:35] dgp when I get back to full time, perhaps I can offer a complet example
Leaving this open since there's doc improvement likely to be useful.