Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Dan Mueth wrote:
> 
> I don't think it is at all possible, or even desireable, to have each
> portion of the database broken into 1000 files.

... then it means that you will force the update of 1000 SPEC files in
each distribution :-(.

>  We will have a handful of
> database file types, so this would mean something like an extra 5000 files
> installed.  It would largely defeat the purpose of having the database
> generated in advance, since scrollkeeper would have to figure out how to
> piece these together every time it is run by the help browser.  For
> example, if the help browser wants the contents list (which is sorted by
> subject), scrollkeeper would have to read all 1000 files, look at the
> contents list categorization description, and then sort and merge all the
> entries.  This would be a big performance problem - every time you open
> your help browser it has to do this over.

... unless there is a caching mechanism that rebuilds the list only when
it is needed (for example each time the number of files in the database
directory has changed). Not easy though.

>  Further, if we ever change the
> format of any of these files at all, having all the work done on the
> builder's machine will guarantee that everybody's system will have file
> version incompatibilities (some packages were built with the old
> scrollkeeper).

This is a generic problem with distributions. I'm facing it for example
with the docbook-tools. A file built on some system often has problems
to run on another system. Basically people assume that once you have a
distribution installed on your machine then you must download any
further software on the same distribution's sites.

What you say here is independant from having 1 or 1000 files in the
database: whether you like it or not, "make install" will be run on the
packager's machine, not on the user's machine. You have to be very
conscious of this.

>  If it is all done on the users machine, then when you
> upgrade scrollkeeper from version 0.X to 0.X+1, a script can run to
> migrate the database if necessary between the old format to the new one.
> I think we are almost guaranteed to want to do this.  While it would be
> nice to design our database files perfectly in our first shot, I really
> don't expect this to happen.

You cannot assume that "make install" is run on the end user's machine.
I think you are starting to see the consequences of this simple fact
only now ;-).

> In fact, we even have problems with adding features.  Suppose
> scrollkeeper-0.2 has a new feature, say searching, with a new database.
> When you upgrade to version 0.2, scrollkeeper should have an install
> script which generates the new database using its list of all the OMF
> files and docs on the system.  Suddenly, the user now has searching
> ability with a complete database :)  If we left all of this work up to the
> build machines, the users would have to wait many months until most
> builders decide to upgrade their scrollkeeper to 0.2 and rebuild the RPMs
> for you.  Then you have to get new versions of each of these 1000
> packages before you can use the new feature (searching).

This is a problem of updates any tool has with distributions.
- either the user gathers his own distribution himself and compiles and
"make install" himself
- either he becomes dependant of the quality of the work of the
packager. Nothing prevents a distribution from using outdated tools.

> Basically, we want each application to install N docs, install between 1
> and N OMF files, and run a script.

If I understand well, "install" for docs and OMF files means only
copying, and therefore it's okay.

The problems start to arise when you try using a tool that modifies a
database. You will have to add at least a %Post section and a %Postun
section. I can foresee distributions saying "This OMF thingie is nice,
but it's too much work for our packagers, and it will slow down even
more our post-installation phase. Let's forget about it." You can assume
that developpers will like it and at lines to heir "make install". But I
wouldn't bet on packagers' enthusiasm. Maybe I'm too pessimistic, but...

>  The docs and OMF files which are
> installed are used by scrollkeeper but not modified after
> installation.  They serve as reference for scrollkeeper to build its
> database from, but do not directly comprise the database.  At install
> (ie. %Post), scrollkeeper clearly needs to use these two files to add info
> to its database files.  After this time, they aren't used
> that much by scrollkeeper.  I think we will probably want something
> analogous to the --rebuilddb feature to RPM which will allow scrollkeeper
> to use its simple list of OMF files on the system and the OMF files
> themselves to rebuild all of the database files.  Presumably this is only
> necessary if the database files are accidentally harmed ("darn - I guess I
> shouldn't have hit ctrl-C during an install").  The original files will
> probably be used sometimes when when scrollkeeper is upgraded to extract
> new information which wasn't needed in previous databases but is for the
> new database. (eg. when new versions of scrollkeeper have new
> functionality)

The only opportunity to avoid having a %Post and a %Postun are to ask
for a manual "--rebuilddb" (ush - everyone will forget that when he/she
installs or uninstalls software) or to have it rebuilt automatically
when some caching mechanism detects that the database is too old.

> > This was misunderstanding from me, I thought the OMF files were
> > concatenated. Forget about the OMF files. If they are just copied "as
> > is" then it's perfect.
> 
> Yes.  We want to copy them "as is", but then use them with a postinstall
> script which modifies the real database files under /var.

This is the problematic part.

> I think we want something along the lines of (b). 'make install:' installs
> the doc(s) and omf file(s). Then it calls scrollkeeper-install, pointing
> it to the doc(s) and omf file(s) it installed and giving it the path
> $prefix/var/scrollkeeper.  If it is an RPM install, this path is something
> like /tmp/scrollkeeper-0.1/var/scrollkeeper.

Just a technical detail: not really. There must be provisions in the
software to make transparent the fact that during build it was in /tmp.

>  If it is a real install it
> is /var/scrollkeeper.  We have a couple choices on how
> scrollkeeper-install works.  I think the easiest is that if the database
> doesn't exist, scrollkeeper just exits.

It will never exist before: remember, you are in tmp/.

>  In this case for a person
> building an RPM, scrollkeeper-install looks for its main list,
> /tmp/scrollkeeper-0.1/var/scrollkeeper/scrollkeeper-docs.  This file
> doesn't exist, so scrollkeeper-install just exits.  If 'make install:' is
> run by hand, scrollkeeper-install wants to append an entry to
> /var/scrollkeeper/scrollkeeper-docs which exists, so scrollkeeper-install
> proceeds happily updating the various database files in /var/scrollkeeper.
> We will need $POST to have scrollkeeper-install.
> 
> Another option would be to generate fresh database files under
> /tmp/scrollkeeper-0.1/var/scrollkeeper if they don't exist already, but I
> really don't see any point in doing this since they won't be installed and
> they will be deleted a couple seconds later.

No, it would be the "usual" way of doing it: the freshly created
database holds the partial (= only for this package) information and
it's this partial database that is merged at postinstallation's time on
the end-user's hard disk. Or even better, this partial database is only
copied but then we have to cope with a database split into as many files
as there are packages with OMF information.

> > Even if you could, it would be a bad idea. A "make install" instruction
> > is supposed to be run equally from the end user's command line and from
> > a packager's spec file.
> 
> I'm not sure exactly what this means.  Does my paragraph above satisfy
> this requirement?  If not, what would need to be changed?  Certainly it
> would not be required that the real databases are changed.

Well, as far as scrollkeeper exits without producing an error code
different than 0 when trying to "make install" if the database does not
exist previously, then yes, you can assume that it can do whatever it
wants, either not create a fresh one or create it. But it must also
behave the same way on the user's hard disk if the database did not
exist previously.

"make install" must behave the same way on the user's hard disk and on
the packager's hard disk. If the database does not exist previously and
if it exits then with an error code, then we have a problem.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/