Re: [Scrollkeeper-devel] Scrollkeeper delay

[Dan M. <mu...@al...>]

On 21 Jun 2003, Malcolm Tredinnick wrote:

> On Sat, 2003-06-21 at 00:52, Dan Mueth wrote:
> > On 20 Jun 2003, Malcolm Tredinnick wrote:
> > 
> > > On Fri, 2003-06-20 at 21:09, Pablo Fernandez wrote:
> > 
> > > > Is there anyway to avoid the rebuild of the database without rebuilding
> > > > the packages that use the scrollkeeper-update program?
> > > 
> > > Not easily. An ugly, but possibly workable solution would be to
> > > temporarily alias the scrollkeeper-update name to something harmless
> > > like ':' and then run scrollkeeper-update once as part of some
> > > post-install fixup.
> > 
> > Is the system really doing a complete rebuild of the database?  If so, 
> > then it is duplicating work and it would make sense to have a way to turn 
> > scrollkeeper off (such as temporarily replacing the binary or having a 
> > line in /etc/scrollkeeper.conf).  Then when the distribution installation 
> > is done, you should update the database.
> 
> I was thinking about this while falling asleep last night and realised a
> solution might be to have an environment variable that prevented a
> database rebuild in a big install scenario. So if
> SCROLLKEEPER_NO_REBUILD is set in the environment, scrollkeeper-rebuild
> just exits when called without doing anything. This can the be disabled
> an re-enabled at will and doesn't require hacking any files . The latter
> case can lead to inconsistencies if the install failed or the machine
> crashed or something, whereas an environment variable would just
> harmlessly disappear.

That is far more elegant :)

I'm not particularly familiar with how installations are typically done.  
It seems completely reasonable that one is free to set environment 
variables and pass them along as needed.  Does anybody know if that is 
right?

> > The real solution to this problem is to extend ScrollKeeper so that one 
> > can process the docs at build time and install the generated data so that 
> > ScrollKeeper doesn't have to do it at install time.
> 
> Can you explain this a bit more, because I am stupid and do not
> understand. :-(
> 
> We already run scrollkeeper-preinstall in the build phase and
> scrollkeeper-update in the install phase. So it sounds like you are
> proposing some kind of optimised output from the build phase so that the
> update part will be faster at install time, but, like I said, I don't
> really understand.

When we build a package, ScrollKeeper is actually run in a way that it 
generates the metadata files from the document and writes them to files.  
However, these files are not kept in the package.  They are just left in 
the temporary build tree and then are later destroyed.  The reasoning is: 
(1) We want to generate and install (in the temporary path) the metadata 
to make sure any errors are found, and (2) We don't keep and install the 
generated metadata because you have to worry about compatibility between 
different versions of ScrollKeeper and potential customization of 
stylesheets on different systems.  At install time, the metadata is 
re-generated from the original documents, which is why ScrollKeeper makes 
package installation time grow dramatically for packages with long 
documents.

The original hope was that we would win the speed race... that libxml and
computers would both speed up faster than the documentation body grows and
that the package installation time would not be too painfully lengthened.  
Since this is not the case, we need to consider the other options which we
originally wanted to avoid (things like pre-generating and installing the
metadata, using cron to process metadata at a more convenient time,
backgrounding the metadata generation, etc.).  They all have their 
drawbacks, but it sounds like we need to provide at least an optional 
workaround so that distributions have a way to avoid lengthening 
distribution installation times significantly.

As for scrollkeeper-preinstall, it is simply a little utility which 
modifies an OMF entry to include the actual path that a document will be 
installed at, since this is generally not known until build time.  It 
doesn't actually read or process the document itself.

-Dan