From: SourceForge.net <no...@so...> - 2012-04-24 15:56:21
|
Feature Requests item #3519865, was opened at 2012-04-20 08:46 Message generated for change (Comment added) made by basilik99 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: 39. Package Manager Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Etienne Basilik (basilik99) Assigned to: Don Porter (dgp) Summary: [package require] is too slow Initial Comment: Hi, As we find at multiple places on the Net, a problem with Tcl is the slowness of [package require] commands. My Tcl distribution is on a network drive and I cannot change that. Here is my loading time: % time {package require tdom; package require md5} 8310807 microseconds per iteration And I also computed 8 seconds on my watch. Very slow... For small scripts that run often, this is big problem. It tends developpers to avoid using any packages. Tclx is a good example: re-code yourself its simple functions if you can to avoid its long loading time! I searched the Internet to see if it could be possible to load packages in background, i.e. when the main Tcl code is interrupted by the OS (during [glob] or [read] commands, or anything that makes the OS interrupt our process). I didn't find anything. I tried the [after] command, but that is not true multiprocessing so [after idle] command is not executed while accessing the I/Os. I would like them to be loaded in background for situations where I don't need these packages right away. I could start doing work while the packages load, then wait for the [package require] cpmpletion and then use the package commands. The overral script time would be smaller. I looked a bit at the Thread package. But I don't think a children thread can load a package for the main program. I doubt that the tdom package for example will be visible in the main script. I also looked at the threaded version of tclsh (base-tcl8.6-thread-win32-ix86.exe), but I didn't understand its benefits... Is there a solution to this problem? Is loading a package into another process possible? Thanks, Etienne ---------------------------------------------------------------------- >Comment By: Etienne Basilik (basilik99) Date: 2012-04-24 08:56 Message: Hi, I use Tcl 8.6.0.0.b6. dgp said: "Trying to convert [package require] to something event-driven is a contradiction in terms" I didn't meant running [package require] in the event loop (and then waiting for its completion using update or vwait). My first idea was really about it running in a different thread, but at the level of Tcl core. Again, when Tcl core commands asks information to the OS (Windows in my case), the tclsh process is suspended while the system call executes. That time could be used by the Tcl core to terminate non-I/O related commands such as [package require]. Now, for the ideas posted, here is the time for loading tdom package alone: % time {package require tdom} 3294262 microseconds per iteration Then I tried what you recommended (using load/source directly): % set dir {some_directory/lib/teapot/package/win32-ix86/lib/tdom0.8.3} % time {load [list [file join $dir tdom083.dll]]; source [list [file join $dir tdom.tcl]]} 2260355 microseconds per iteration This is saving 1 second. Its good, but not enough. It does not worth hardcoding the paths. I also tried to do both [package require] in a separate thread: package require Thread set tdom_md5_thread [thread::create] thread::send -async $tdom_md5_thread {package require tdom} thread::send -async $tdom_md5_thread {package require md5} ... search in directories here while the 2 [package require] are performed ... thread::send -async $tdom_md5_thread "set file_to_check $file_to_check" thread::send -async $tdom_md5_thread \ { set xrd_in [open $file_to_check r] dom parse -simple [read $xrd_in] xrd_dom_tree close $xrd_in set xrd_first_title0_node [lindex [$xrd_dom_tree selectNodes {XROD//APIDOC//TITLE[@LEVEL='0']}] 0] return "[$xrd_first_title0_node asText].ptu" } ptu_name ... some code ... vwait ptu_name It works, but I doubt that I am really gaining time. One reason is the time to initialize the thread: % time {package require Thread} 1633521 microseconds per iteration % time {thread::create} 972502 microseconds per iteration But also this complexifies my script. Anyway, thanks for posting this idea as it may be useable in other situations. Etienne ---------------------------------------------------------------------- Comment By: Don Porter (dgp) Date: 2012-04-23 08:26 Message: Revised subject and Category to the actual complaint. Trying to convert [package require] to something event-driven is a contradiction in terms. A command that did something like that would be different enough to need a new name. You don't report what release of Tcl delivers these results. If you're not on at least 8.5.11, get there. Then conversion of the most critical packages to TM format is one tool available. Another idea is 680169. ---------------------------------------------------------------------- Comment By: Serg G. Brester (sebres) Date: 2012-04-21 15:07 Message: Try to load the package direct, without "package require"... Read package index information from pkgIndex file and load corresponding commands directly ("source ..." for tcl or "load ..."). So you could save the time for glob. ---------------------------------------------------------------------- Comment By: Twylite (twylite) Date: 2012-04-20 09:48 Message: Scripts are loaded into an interp, and an interp is never shared across threads. This means that execution within an interp is effective single-threaded, and (as you surmised) a child thread/interp cannot load a package for another interp. If your interp is not busy then you can load packages "in the background" using [after idle], but this won't help you if the interp is running flat-out and never enters the event loop. You could load packages into an interp in a separate thread, and use thread::send to call commands in those packages, depending on your application's requirements. With regards to the slow package loading, I have also noticed and been investigating this problem. A significant contributing factor is the use of [glob] within the package/module discovery logic. Anti-virus applications can also dramatically reduce the performance of package loading. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894 |