From: SourceForge.net <no...@so...> - 2012-04-25 13:46:05
|
Feature Requests item #3519865, was opened at 2012-04-20 08:46 Message generated for change (Comment added) made by dkf You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: 39. Package Manager Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Etienne Basilik (basilik99) Assigned to: Don Porter (dgp) Summary: [package require] is too slow Initial Comment: Hi, As we find at multiple places on the Net, a problem with Tcl is the slowness of [package require] commands. My Tcl distribution is on a network drive and I cannot change that. Here is my loading time: % time {package require tdom; package require md5} 8310807 microseconds per iteration And I also computed 8 seconds on my watch. Very slow... For small scripts that run often, this is big problem. It tends developpers to avoid using any packages. Tclx is a good example: re-code yourself its simple functions if you can to avoid its long loading time! I searched the Internet to see if it could be possible to load packages in background, i.e. when the main Tcl code is interrupted by the OS (during [glob] or [read] commands, or anything that makes the OS interrupt our process). I didn't find anything. I tried the [after] command, but that is not true multiprocessing so [after idle] command is not executed while accessing the I/Os. I would like them to be loaded in background for situations where I don't need these packages right away. I could start doing work while the packages load, then wait for the [package require] cpmpletion and then use the package commands. The overral script time would be smaller. I looked a bit at the Thread package. But I don't think a children thread can load a package for the main program. I doubt that the tdom package for example will be visible in the main script. I also looked at the threaded version of tclsh (base-tcl8.6-thread-win32-ix86.exe), but I didn't understand its benefits... Is there a solution to this problem? Is loading a package into another process possible? Thanks, Etienne ---------------------------------------------------------------------- >Comment By: Donal K. Fellows (dkf) Date: 2012-04-25 06:46 Message: I was suggesting home dir because that's usually writable. (Note, if not – e.g., because the current user doesn't have an existing home dir, which is a situation I'm in with software deployed on my servers — then simply don't cache.) If there's a more appropriate location on Windows, then please read as if I was suggesting the use of that instead. There's lots of fine details about Win that I don't know. :-) Can't help about [glob] within pkgIndex.tcl (there's too much complexity in pkgIndex.tcl files in the wild for us to consider parsing them by any mechanism other than evaluation) but I think it would be possible to replace the TM searcher with this scheme too; I think the cache coherency condition would still be the same. The difference is that TMs synthesize the pkg-index info from the [glob] results rather than looking for a specific file and sourcing that. I'd be tempted to ignore tclIndex from this scheme, given that it doesn't normally exist. At least for a first hack. I'd be interested to see how things get on. ---------------------------------------------------------------------- Comment By: Twylite (twylite) Date: 2012-04-25 06:30 Message: @dkf: You said "Let the cache be a dictionary, indexed by normalized filename with one entry per dir seen in $auto_path (or whatever path is used), stored in the user's home directory" I think a user-and-machine-specific cache is key, so this sounds to me like the right direction. Using one file per auto_path dir is an interesting approach - I'll look at doing some performance profiling on it. Some comments: - "Home directory" is fine on *nix, but needs to be local appdata path on Windows. It is common on Active Directory domains for the home directory to be on a network share (in which case accessing the cache will be slow, and the cache will be visible to different machines). - The cache you suggest will avoid searching for and loading pkgIndex files, but will not address the [glob]s done within pkgIndex files or tcl::tm::unknownHandler, nor the [file exists] checks for tclIndex. I have been looking into caching [package ifneeded] calls, and overriding [package unknown] to be a nop (or empty the auto_path and module paths) if the cache is up to date. It is an approach that will avoid all [glob] and [file exists] calls, as well as not loading pkgIndex files. It would be trivial to extend this to cover tclIndex files as well. Using your auto_path and [file mtime] ideas one could improve the efficiency with which such a cache could be rebuilt. Do you see any problems with this approach? ---------------------------------------------------------------------- Comment By: Donal K. Fellows (dkf) Date: 2012-04-25 05:42 Message: Also, I've good reason to believe that this scheme would work fine with ActiveTcl and the (saner) Linux distributions. :-) ---------------------------------------------------------------------- Comment By: Donal K. Fellows (dkf) Date: 2012-04-25 05:40 Message: If we assume that the contents of a package don't change, just that new packages are installed (each in their own directory tree) then we *can* get a decision on whether to rebuild the cache. Let the cache be a dictionary, indexed by normalized filename with one entry per dir seen in $auto_path (or whatever path is used), stored in the user's home directory (or below; if the user's home directory doesn't exist or isn't writable, we just don't cache; similarly, in a safe interp we don't use the cache either, or at least not unless someone figures out how to make a safe version of it). Then all we need to do is to store the [file mtime] of the dir and compare; if it changes, that means something's been added or removed and we should rebuild. The cache contents should be tuples of exact-normalized-filename-of-pkgIndex.tcl and contents-of-that-pkgIndex.tcl; that would be enough to allow us to synthesize sourcing the file without having to search for it and actually source it. That *should* be enough to boost loading. Users who insist on having /usr/lib in their auto_path get what's due to them for having such ill-considered installations. Probably ought to also have a command to allow flushing of the whole cache or just a part of it. Just in case someone needs something other than what can be done conveniently. Open question: does [file mtime $pathEntry] work suitably for this on Windows? (NB: remember that not all entries on the auto_path exist.) Inspiration code: foreach d $auto_path {if {[file exists $d]} { puts [file normalize $d]:[clock format [file mtime $d]] }} ---------------------------------------------------------------------- Comment By: Donal K. Fellows (dkf) Date: 2012-04-25 05:19 Message: It's easy to cache such things. Remember, we can also cache the place where the flle was found, and there's [info script $value]. The problem is how to detect when the cache needs to be rebuilt. (There's also the issue of *where* to maintain the cache and *who* will maintain it.) ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-25 05:13 Message: 1. That is why there is actually variable "dir", and you can load custom pkgIndex.tcl that used info script with source. How do you cache such dinamically pckIndex files? 2. And? See source code: this rescan will load only NEW dirs. This is already "cached". 3. I see that not a problem. 4. :) Yes, but we will not forbid [glob] command here... We have this "combine in one" solution already few years and it work for all "makeIndex" packages, that are auto created. My global pckIndex have currentlly 850 packages. My global tclIndex have more than 1000 namespaces. So instead of 850+1000 files we load 2 only. ---------------------------------------------------------------------- Comment By: Twylite (twylite) Date: 2012-04-25 04:46 Message: It really _is_ a caching problem: 1. Each pkgIndex.tcl is a proper Tcl script and can take actions that are specific to the absolute or relative location of the script, or to the environment. It is not possible - without imposing unenforceable rules on what pkgIndex.tcl can do - to automate the combination of multiple pkgIndex.tcl files. A simple [file dirname [info script]] - common in pkgIndex files - won't work in a merged script without an emulation layer to Do The Right Thing. 2. Various scripts modify auto_path, or use catch { package require somePkg } to attempt to load package that may or may not be present. Either will cause a rescan for pkgIndex files. 3. Nothing is stopping a pkgIndex from using [glob] to discover files for which to issue [package ifneeded] commands. I've been using this sort of "magic indexing" to do something akin to Tcl Modules, long before there were Tcl Modules. 4. Merging pkgIndex files doesn't avoid the [glob] calls in ::tcl::tm::UnknownHandler. Combining multiple pkgIndex files into one can work in simple cases, and can result in a limited performance improvement in simple cases. But it is not a general solution, and it doesn't work for every package out there. It is certainly possible to build a better makeIndex with a different packaging mechanism, but that doesn't solve the problem for developers who are experiencing slow startup performance with Tcl _right now_. ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-25 03:47 Message: This is not really a caching... It's another makeIndex feature. The trick is to disolve many pckIndex.tcl with only ONE or FEW pckIndex.tcl and tclIndex (that included all other system wide), to shrink auto_path list. You should not forget that all pckIndex.tcl (from auto_path) will be loaded by first use of [package require, package present etc]. And I do not want also this feature in tcl core, because to abstract resp. system dependent and each developer could create such own makeIndex ín few minutes. So I have my own solution for tcl and for tk different on windows and linux, that customized for my purposes. ---------------------------------------------------------------------- Comment By: Twylite (twylite) Date: 2012-04-25 03:22 Message: Effectively sebres is talking about caching. This can be done, and I've done some investigation, but it can be tricky to right for all environments. Background: I use Tcl on a variety of systems including an Atom-based notebook (Windows 7 32-bit) with a 4500rpm HDD. tclsh loads quickly on the notebook, but wish86t.exe takes upwards of 25 seconds to display a console! I have a company project that uses Wub and a bunch of our own packages, and can take over 110 seconds to load. I have found a number of performance killers: (1) we're using a trofs VFS; trofs itself isn't slow but each new source/open causes a CreateFile() which gets the anti-virus involved. (2) missing packages cause a re-[glob] of all library paths (profile [package require ip] from tcllib on a system that doesn't have md5 or critcl if you want to see the effect). (3) [glob] is slow on slow drives (including USB disks, network drives, etc). A caching solution could monitor and cache calls to [package ifneeded], and the [glob] calls done to locate Tcl modules. This would avoid the need to use [glob] and [file exists] in most cases, and improve performance. The catch is that the cache is specific to both the Tcl installation, and the environment in which it is invoked. Two different users on the same PC, or two different users running a network install of Tcl, require different caches. The best approach I know of at the moment is to have a VERSION file in the install (which _must_ be changed if the install changes), and a cache file stored in the users local profile on any given machine. If the local cache is missing or disagrees with the VERSION file, then the cache must be regenerated. I _think_ this handles the problem for Windows. I don't know if it can adapted to improve the performance of portable installs. And I don't know the impact on non-Windows systems. ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-25 01:32 Message: You have probably not correctly understood it with [package require]. Don't need remove [package require] anywhere from source code. You could change package handling so that does not search pkgIndex.tcl everywhere. So for example with preseting [package ifneeded]. Or you can for example make ONE pkgIndex.tcl (as concatenation of all other with set of variable dir) and place it ex. in tcl-inst-dir/lib/globalIndex/pkgIndex.tcl. The same you should make with the tclIndex (auto loading). In init.tcl you can set auto_path alone to the "tcl-inst-dir/lib/globalIndex". Or just make your own makeIndex batch, that do this. Ex. for globalIndex/pkgIndex.tcl: set globdir [info library] ## http: set dir $globdir/http if {![package vsatisfies [package provide Tcl] 8.4]} {return} package ifneeded http 2.7.8 [list tclPkgSetup $dir http 2.7.8 {{http.tcl source {::http::config ::http::formatQuery ::http::geturl ::http::reset ::http::wait ::http::register ::http::unregister ::http::mapReply}}}] ## msgcat: set dir $globdir/msgcat if {![package vsatisfies [package provide Tcl] 8.5]} {return} package ifneeded msgcat 1.4.4 [list source [file join $dir msgcat.tcl]] .... ## next pkgIndex... .... That will completelly eliminate overhead for [glob] to search pkgIndex.tcl and/or tclIndex. ---------------------------------------------------------------------- Comment By: Etienne Basilik (basilik99) Date: 2012-04-24 14:29 Message: Hi, Indeed, I could install Tcl on my local drive rather than on a network drive as I actually do. My colleague did a test and divides loading time by 8. But we are multiple developers and we have many tools pointing to the Tcl install dir on the network drive. This approach helps centralizing and provides a unified Tcl version for everyone. When choosing the programming language for a new project, I now ask myself the question: "Is this a small program that will be run many times?" If so, I tend to avoid using Tcl because of this long startup latency. Don't you think this kind of problem should be solved in future Tcl versions by introducing a new way of loading packages? The libraries are installed in static directories. This never changes except when you install new ones via teapot. Doesn't it seem strange that Tcl has to search again and again for them each time an interpreter is created? That search could be done once and for all when no new libraries are installed. Or, package loading may be modified to avoid this search. Standard directory names could be used for example to help Tcl get right to the package directory without searching. I never did Tcl development, so I don't know what's feasible and what's not. I am just expressing a user's need. Thanks for your help, Étienne ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2012-04-24 11:36 Message: If you don't need a package right away, you can save your [package require] for the time when you do need it. However, you should know that that won't really help unless the [package require]s that you do need right away can all be satisfied by packages in the Tcl Module (TM) format. As soon as you have to go off seeking into the pkgIndex.tcl wilds, you are going to pay full price. That's how [tclPkgUnknown] was designed, and most packages in existence have crafted their installation patterns to follow that design. One other practical thing you could do would be to install your Tcl somewhere other than its default location in /usr so that you aren't searching big system directories like /usr/lib by default. ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2012-04-24 11:31 Message: No, you're way off base. The architecture of [package] doesn't permit anything like what you are suggesting. ---------------------------------------------------------------------- Comment By: Etienne Basilik (basilik99) Date: 2012-04-24 11:18 Message: Hi sebres, You are right: tdom.tcl contains a 2 other [package require]. But, I can't do nothing about it... Are you suggesting to change that file eventhough it is a library file? I doubt skipping [package require] is the way to go... Don't you think so? Howver (and this is also an answer to dgp), we could imagine a new "package prefetch <package_name>" command which would mean "I don't need that package right away but start loading it.". Then "package require" semantic would change a bit to become: "Now, I really need that package so if loading it is not still complete (or hasn't started), freeze the execution until it is." An even better solution for the final user would be that [package require] itself would start the prefetch and Tcl would complete the prefetch/load when it encounters the first command from that package (command "dom" in my case). But I think this solution is too complex to be implemented... Note: if we are able to spawn a [package require] rather than run it in blocking mode, it may also enhance the opening speed of tclsh.exe... Thanks, Étienne ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-24 09:21 Message: You should completelly exclude command "package require". Be sure that this will be never executed. Because some tcl scripts (from packages), could load another packages and so on. Ex.: tdom.tcl required tdom and uri packages (that call again "glob" over you net share). At first do something like this: >>> proc package {args} { if {[lindex $args 0] == "require"} { error "Prohibited: $args" } package {*}$args } <<< And so exclude each package require. It is possible that it is not only "package require". Have you or could you build any profiler to log an exec time of each command? ---------------------------------------------------------------------- Comment By: Etienne Basilik (basilik99) Date: 2012-04-24 08:56 Message: Hi, I use Tcl 8.6.0.0.b6. dgp said: "Trying to convert [package require] to something event-driven is a contradiction in terms" I didn't meant running [package require] in the event loop (and then waiting for its completion using update or vwait). My first idea was really about it running in a different thread, but at the level of Tcl core. Again, when Tcl core commands asks information to the OS (Windows in my case), the tclsh process is suspended while the system call executes. That time could be used by the Tcl core to terminate non-I/O related commands such as [package require]. Now, for the ideas posted, here is the time for loading tdom package alone: % time {package require tdom} 3294262 microseconds per iteration Then I tried what you recommended (using load/source directly): % set dir {some_directory/lib/teapot/package/win32-ix86/lib/tdom0.8.3} % time {load [list [file join $dir tdom083.dll]]; source [list [file join $dir tdom.tcl]]} 2260355 microseconds per iteration This is saving 1 second. Its good, but not enough. It does not worth hardcoding the paths. I also tried to do both [package require] in a separate thread: package require Thread set tdom_md5_thread [thread::create] thread::send -async $tdom_md5_thread {package require tdom} thread::send -async $tdom_md5_thread {package require md5} ... search in directories here while the 2 [package require] are performed ... thread::send -async $tdom_md5_thread "set file_to_check $file_to_check" thread::send -async $tdom_md5_thread \ { set xrd_in [open $file_to_check r] dom parse -simple [read $xrd_in] xrd_dom_tree close $xrd_in set xrd_first_title0_node [lindex [$xrd_dom_tree selectNodes {XROD//APIDOC//TITLE[@LEVEL='0']}] 0] return "[$xrd_first_title0_node asText].ptu" } ptu_name ... some code ... vwait ptu_name It works, but I doubt that I am really gaining time. One reason is the time to initialize the thread: % time {package require Thread} 1633521 microseconds per iteration % time {thread::create} 972502 microseconds per iteration But also this complexifies my script. Anyway, thanks for posting this idea as it may be useable in other situations. Etienne ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2012-04-23 08:26 Message: Revised subject and Category to the actual complaint. Trying to convert [package require] to something event-driven is a contradiction in terms. A command that did something like that would be different enough to need a new name. You don't report what release of Tcl delivers these results. If you're not on at least 8.5.11, get there. Then conversion of the most critical packages to TM format is one tool available. Another idea is 680169. ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-21 15:07 Message: Try to load the package direct, without "package require"... Read package index information from pkgIndex file and load corresponding commands directly ("source ..." for tcl or "load ..."). So you could save the time for glob. ---------------------------------------------------------------------- Comment By: Twylite (twylite) Date: 2012-04-20 09:48 Message: Scripts are loaded into an interp, and an interp is never shared across threads. This means that execution within an interp is effective single-threaded, and (as you surmised) a child thread/interp cannot load a package for another interp. If your interp is not busy then you can load packages "in the background" using [after idle], but this won't help you if the interp is running flat-out and never enters the event loop. You could load packages into an interp in a separate thread, and use thread::send to call commands in those packages, depending on your application's requirements. With regards to the slow package loading, I have also noticed and been investigating this problem. A significant contributing factor is the use of [glob] within the package/module discovery logic. Anti-virus applications can also dramatically reduce the performance of package loading. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894 |