[Tcl-bugs] [ tcl-Feature Requests-3519865 ] [package require] is too slow

[SourceForge.net <no...@so...>]

Feature Requests item #3519865, was opened at 2012-04-20 08:46
Message generated for change (Comment added) made by sebres
You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894

Please note that this message will contain a full copy of the comment thread,
including the initial issue submission, for this request,
not just the latest update.
Category: 39. Package Manager
Group: None
Status: Open
Resolution: None
Priority: 5
Private: No
Submitted By: Etienne Basilik (basilik99)
Assigned to: Don Porter (dgp)
Summary: [package require] is too slow

Initial Comment:
Hi,

As we find at multiple places on the Net, a problem with Tcl is the slowness of [package require] commands. My Tcl distribution is on a network drive and I cannot change that. Here is my loading time:

% time {package require tdom; package require md5}
8310807 microseconds per iteration

And I also computed 8 seconds on my watch. Very slow... For small scripts that run often, this is big problem. It tends developpers to avoid using any packages. Tclx is a good example: re-code yourself its simple functions if you can to avoid its long loading time!

I searched the Internet to see if it could be possible to load packages in background, i.e. when the main Tcl code is interrupted by the OS (during [glob] or [read] commands, or anything that makes the OS interrupt our process). I didn't find anything. I tried the [after] command, but that is not true multiprocessing so [after idle] command is not executed while accessing the I/Os.

I would like them to be loaded in background for situations where I don't need these packages right away. I could start doing work while the packages load, then wait for the [package require] cpmpletion and then use the package commands. The overral script time would be smaller.

I looked a bit at the Thread package. But I don't think a children thread can load a package for the main program. I doubt that the tdom package for example will be visible in the main script.

I also looked at the threaded version of tclsh (base-tcl8.6-thread-win32-ix86.exe), but I didn't understand its benefits...

Is there a solution to this problem? Is loading a package into another process possible?

Thanks,
Etienne

----------------------------------------------------------------------

>Comment By: Serg G. Brester (sebres)
Date: 2012-04-25 05:13

Message:
1. That is why there is actually variable "dir", and you can load custom
pkgIndex.tcl that used info script with source. How do you cache such
dinamically pckIndex files?
2. And? See source code: this rescan will load only NEW dirs. This is
already "cached".
3. I see that not a problem.
4. :) Yes, but we will not forbid [glob] command here...
We have this "combine in one" solution already few years and it work for
all "makeIndex" packages, that are auto created. My global pckIndex have
currentlly 850 packages. My global tclIndex have more than 1000 namespaces.
So instead of 850+1000 files we load 2 only.

----------------------------------------------------------------------

Comment By: Twylite (twylite)
Date: 2012-04-25 04:46

Message:
It really _is_ a caching problem:

1.  Each pkgIndex.tcl is a proper Tcl script and can take actions that are
specific to the absolute or relative location of the script, or to the
environment.  It is not possible - without imposing unenforceable rules on
what pkgIndex.tcl can do - to automate the combination of multiple
pkgIndex.tcl files.  A simple [file dirname [info script]] - common in
pkgIndex files - won't work in a merged script without an emulation layer
to Do The Right Thing.

2. Various scripts modify auto_path, or use catch { package require somePkg
} to attempt to load package that may or may not be present.  Either will
cause a rescan for pkgIndex files.

3. Nothing is stopping a pkgIndex from using [glob] to discover files for
which to issue [package ifneeded] commands.  I've been using this sort of
"magic indexing" to do something akin to Tcl Modules, long before there
were Tcl Modules.

4. Merging pkgIndex files doesn't avoid the [glob] calls in
::tcl::tm::UnknownHandler.

Combining multiple pkgIndex files into one can work in simple cases, and
can result in a limited performance improvement in simple cases.  But it is
not a general solution, and it doesn't work for every package out there.

It is certainly possible to build a better makeIndex with a different
packaging mechanism, but that doesn't solve the problem for developers who
are experiencing slow startup performance with Tcl _right now_.


----------------------------------------------------------------------

Comment By: Serg G. Brester (sebres)
Date: 2012-04-25 03:47

Message:
This is not really a caching... It's another makeIndex feature.
The trick is to disolve many pckIndex.tcl with only ONE or FEW pckIndex.tcl
and tclIndex (that included all other system wide),
to shrink auto_path list.
You should not forget that all pckIndex.tcl (from auto_path) will be loaded
by first use of [package require, package present etc]. 
And I do not want also this feature in tcl core, because to abstract resp.
system dependent and each developer could create such own makeIndex ín few
minutes. 
So I have my own solution for tcl and for tk different on windows and
linux, that customized for my purposes.

----------------------------------------------------------------------

Comment By: Twylite (twylite)
Date: 2012-04-25 03:22

Message:
Effectively sebres is talking about caching.  This can be done, and I've
done some investigation, but it can be tricky to right for all
environments.

Background: I use Tcl on a variety of systems including an Atom-based
notebook (Windows 7 32-bit) with a 4500rpm HDD.  tclsh loads quickly on the
notebook, but wish86t.exe takes upwards of 25 seconds to display a console!
 I have a company project that uses Wub and a bunch of our own packages,
and can take over 110 seconds to load.

I have found a number of performance killers: (1) we're using a trofs VFS;
trofs itself isn't slow but each new source/open causes a CreateFile()
which gets the anti-virus involved.  (2) missing packages cause a re-[glob]
of all library paths (profile [package require ip] from tcllib on a system
that doesn't have md5 or critcl if you want to see the effect).  (3) [glob]
is slow on slow drives (including USB disks, network drives, etc).

A caching solution could monitor and cache calls to [package ifneeded], and
the [glob] calls done to locate Tcl modules.  This would avoid the need to
use [glob] and [file exists] in most cases, and improve performance.

The catch is that the cache is specific to both the Tcl installation, and
the environment in which it is invoked.  Two different users on the same
PC, or two different users running a network install of Tcl, require
different caches.

The best approach I know of at the moment is to have a VERSION file in the
install (which _must_ be changed if the install changes), and a cache file
stored in the users local profile on any given machine.  If the local cache
is missing or disagrees with the VERSION file, then the cache must be
regenerated.

I _think_ this handles the problem for Windows.  I don't know if it can
adapted to improve the performance of portable installs.  And I don't know
the impact on non-Windows systems.


----------------------------------------------------------------------

Comment By: Serg G. Brester (sebres)
Date: 2012-04-25 01:32

Message:
You have probably not correctly understood it with [package require].
Don't need remove  [package require] anywhere from source code.
You could change package handling so that does not search pkgIndex.tcl
everywhere.
So for example with preseting [package ifneeded].
Or you can for example make ONE pkgIndex.tcl (as concatenation of all other
with set of variable dir) and place it ex. in
tcl-inst-dir/lib/globalIndex/pkgIndex.tcl.
The same you should make with the tclIndex (auto loading).
In init.tcl you can set auto_path alone to the
"tcl-inst-dir/lib/globalIndex".

Or just make your own makeIndex batch, that do this.

Ex. for globalIndex/pkgIndex.tcl:

set globdir [info library]
## http:
set dir $globdir/http
if {![package vsatisfies [package provide Tcl] 8.4]} {return}
package ifneeded http 2.7.8 [list tclPkgSetup $dir http 2.7.8 {{http.tcl
source {::http::config ::http::formatQuery ::http::geturl ::http::reset
::http::wait ::http::register ::http::unregister ::http::mapReply}}}]
## msgcat:
set dir $globdir/msgcat
if {![package vsatisfies [package provide Tcl] 8.5]} {return}
package ifneeded msgcat 1.4.4 [list source [file join $dir msgcat.tcl]]
....
## next pkgIndex...
....

That will completelly eliminate overhead for [glob] to search pkgIndex.tcl
and/or tclIndex.

----------------------------------------------------------------------

Comment By: Etienne Basilik (basilik99)
Date: 2012-04-24 14:29

Message:
Hi,

Indeed, I could install Tcl on my local drive rather than on a network
drive as I actually do. My colleague did a test and divides loading time by
8. But we are multiple developers and we have many tools pointing to the
Tcl install dir on the network drive. This approach helps centralizing and
provides a unified Tcl version for everyone.

When choosing the programming language for a new project, I now ask myself
the question: "Is this a small program that will be run many times?" If so,
I tend to avoid using Tcl because of this long startup latency.

Don't you think this kind of problem should be solved in future Tcl
versions by introducing a new way of loading packages? The libraries are
installed in static directories. This never changes except when you install
new ones via teapot. Doesn't it seem strange that Tcl has to search again
and again for them each time an interpreter is created? That search could
be done once and for all when no new libraries are installed. Or, package
loading may be modified to avoid this search. Standard directory names
could be used for example to help Tcl get right to the package directory
without searching.

I never did Tcl development, so I don't know what's feasible and what's
not. I am just expressing a user's need.

Thanks for your help,
Étienne

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2012-04-24 11:36

Message:
If you don't need a package right away, you can
save your [package require] for the time when you
do need it.

However, you should know that that won't really help
unless the [package require]s that you do need right
away can all be satisfied by packages in the Tcl Module (TM)
format.  As soon as you have to go off seeking into
the pkgIndex.tcl wilds, you are going to pay full price.  That's
how [tclPkgUnknown] was designed, and most packages
in existence have crafted their installation patterns to
follow that design.

One other practical thing you could do would be to 
install your Tcl somewhere other than its default
location in /usr so that you aren't searching big
system directories like /usr/lib by default.

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2012-04-24 11:31

Message:
No, you're way off base.  The architecture of
[package] doesn't permit anything like what
you are suggesting.

----------------------------------------------------------------------

Comment By: Etienne Basilik (basilik99)
Date: 2012-04-24 11:18

Message:
Hi sebres,

You are right: tdom.tcl contains a 2 other [package require]. But, I can't
do nothing about it... Are you suggesting to change that file eventhough it
is a library file? I doubt skipping [package require] is the way to go...
Don't you think so?

Howver (and this is also an answer to dgp), we could imagine a new "package
prefetch <package_name>" command which would mean "I don't need that
package right away but start loading it.". Then "package require" semantic
would change a bit to become: "Now, I really need that package so if
loading it is not still complete (or hasn't started), freeze the execution
until it is."

An even better solution for the final user would be that [package require]
itself would start the prefetch and Tcl would complete the prefetch/load
when it encounters the first command from that package (command "dom" in my
case). But I think this solution is too complex to be implemented...

Note: if we are able to spawn a [package require] rather than run it in
blocking mode, it may also enhance the opening speed of tclsh.exe...

Thanks,
Étienne

----------------------------------------------------------------------

Comment By: Serg G. Brester (sebres)
Date: 2012-04-24 09:21

Message:
You should completelly exclude command "package require". Be sure that this
will be never executed.
Because some tcl scripts (from packages), could load another packages and
so on.
Ex.: tdom.tcl required tdom and uri packages (that call again "glob" over
you net share).

At first do something like this: 
>>>
proc package {args} {
  if {[lindex $args 0] == "require"} {
    error "Prohibited: $args"
  }
  package {*}$args
}
<<<
And so exclude each package require. 

It is possible that it is not only "package require". 
Have you or could you build any profiler to log an exec time of each
command?

----------------------------------------------------------------------

Comment By: Etienne Basilik (basilik99)
Date: 2012-04-24 08:56

Message:
Hi,

I use Tcl 8.6.0.0.b6.

dgp said:
"Trying to convert [package require] to something event-driven is a
contradiction in terms"
I didn't meant running [package require] in the event loop (and then
waiting for its completion using update or vwait). My first idea was really
about it running in a different thread, but at the level of Tcl core.
Again, when Tcl core commands asks information to the OS (Windows in my
case), the tclsh process is suspended while the system call executes. That
time could be used by the Tcl core to terminate non-I/O related commands
such as [package require].

Now, for the ideas posted, here is the time for loading tdom package
alone:
% time {package require tdom}
3294262 microseconds per iteration

Then I tried what you recommended (using load/source directly):
% set dir {some_directory/lib/teapot/package/win32-ix86/lib/tdom0.8.3}
% time {load [list [file join $dir tdom083.dll]]; source [list [file join
$dir tdom.tcl]]}
2260355 microseconds per iteration

This is saving 1 second. Its good, but not enough. It does not worth
hardcoding the paths.

I also tried to do both [package require] in a separate thread:

package require Thread
set tdom_md5_thread [thread::create]
thread::send -async $tdom_md5_thread {package require tdom}
thread::send -async $tdom_md5_thread {package require md5}

... search in directories here while the 2 [package require] are performed
...

thread::send -async $tdom_md5_thread "set file_to_check $file_to_check"

thread::send -async $tdom_md5_thread \
{
   set xrd_in [open $file_to_check r]
   dom parse -simple [read $xrd_in] xrd_dom_tree
   close $xrd_in
   set xrd_first_title0_node [lindex [$xrd_dom_tree selectNodes
{XROD//APIDOC//TITLE[@LEVEL='0']}] 0]
   return "[$xrd_first_title0_node asText].ptu"
} ptu_name

... some code ...
vwait ptu_name

It works, but I doubt that I am really gaining time. One reason is the time
to initialize the thread:
% time {package require Thread}
1633521 microseconds per iteration
% time {thread::create}
972502 microseconds per iteration

But also this complexifies my script. Anyway, thanks for posting this idea
as it may be useable in other situations.

Etienne

----------------------------------------------------------------------

Comment By: Don Porter (dgp)
Date: 2012-04-23 08:26

Message:
Revised subject and Category to the actual complaint.

Trying to convert [package require] to something event-driven
is a contradiction in terms.  A command that did something like
that would be different enough to need a new name.

You don't report what release of Tcl delivers these results.
If you're not on at least 8.5.11, get there.  Then conversion
of the most critical packages to TM format is one tool available.
Another idea is 680169.

----------------------------------------------------------------------

Comment By: Serg G. Brester (sebres)
Date: 2012-04-21 15:07

Message:
Try to load the package direct, without "package require"...
Read package index information from pkgIndex file and load corresponding
commands directly ("source ..." for tcl or "load ..."). So you could save
the time for glob.

----------------------------------------------------------------------

Comment By: Twylite (twylite)
Date: 2012-04-20 09:48

Message:
Scripts are loaded into an interp, and an interp is never shared across
threads.  This means that execution within an interp is effective
single-threaded, and (as you  surmised) a child thread/interp cannot load a
package for another interp.

If your interp is not busy then you can load packages "in the background"
using [after idle], but this won't help you if the interp is running
flat-out and never enters the event loop.

You could load packages into an interp in a separate thread, and use
thread::send to call commands in those packages, depending on your
application's requirements.

With regards to the slow package loading, I have also noticed and been
investigating this problem.  A significant contributing factor is the use
of [glob] within the package/module discovery logic.  Anti-virus
applications can also dramatically reduce the performance of package
loading.

----------------------------------------------------------------------

You can respond by visiting: 
https://sourceforge.net/tracker/?func=detail&atid=360894&aid=3519865&group_id=10894