Re: [Scrollkeeper-devel] What about the distributions?

[Laszlo K. <las...@su...>]

I dont seem to be able to find the mail you replied to in this one. Was
that between Dan and yourself only?

Laszlo

> 
> Dan Mueth wrote:
> >
> > I don't think it is at all possible, or even desireable, to have each
> > portion of the database broken into 1000 files.
> 
> ... then it means that you will force the update of 1000 SPEC files in
> each distribution :-(.
> 
> >  We will have a handful of
> > database file types, so this would mean something like an extra 5000 files
> > installed.  It would largely defeat the purpose of having the database
> > generated in advance, since scrollkeeper would have to figure out how to
> > piece these together every time it is run by the help browser.  For
> > example, if the help browser wants the contents list (which is sorted by
> > subject), scrollkeeper would have to read all 1000 files, look at the
> > contents list categorization description, and then sort and merge all the
> > entries.  This would be a big performance problem - every time you open
> > your help browser it has to do this over.
> 
> ... unless there is a caching mechanism that rebuilds the list only when
> it is needed (for example each time the number of files in the database
> directory has changed). Not easy though.
> 
> >  Further, if we ever change the
> > format of any of these files at all, having all the work done on the
> > builder's machine will guarantee that everybody's system will have file
> > version incompatibilities (some packages were built with the old
> > scrollkeeper).
> 
> This is a generic problem with distributions. I'm facing it for example
> with the docbook-tools. A file built on some system often has problems
> to run on another system. Basically people assume that once you have a
> distribution installed on your machine then you must download any
> further software on the same distribution's sites.
> 
> What you say here is independant from having 1 or 1000 files in the
> database: whether you like it or not, "make install" will be run on the
> packager's machine, not on the user's machine. You have to be very
> conscious of this.
> 
> >  If it is all done on the users machine, then when you
> > upgrade scrollkeeper from version 0.X to 0.X+1, a script can run to
> > migrate the database if necessary between the old format to the new one.
> > I think we are almost guaranteed to want to do this.  While it would be
> > nice to design our database files perfectly in our first shot, I really
> > don't expect this to happen.
> 
> You cannot assume that "make install" is run on the end user's machine.
> I think you are starting to see the consequences of this simple fact
> only now ;-).
> 
> > In fact, we even have problems with adding features.  Suppose
> > scrollkeeper-0.2 has a new feature, say searching, with a new database.
> > When you upgrade to version 0.2, scrollkeeper should have an install
> > script which generates the new database using its list of all the OMF
> > files and docs on the system.  Suddenly, the user now has searching
> > ability with a complete database :)  If we left all of this work up to the
> > build machines, the users would have to wait many months until most
> > builders decide to upgrade their scrollkeeper to 0.2 and rebuild the RPMs
> > for you.  Then you have to get new versions of each of these 1000
> > packages before you can use the new feature (searching).
> 
> This is a problem of updates any tool has with distributions.
> - either the user gathers his own distribution himself and compiles and
> "make install" himself
> - either he becomes dependant of the quality of the work of the
> packager. Nothing prevents a distribution from using outdated tools.
> 
> > Basically, we want each application to install N docs, install between 1
> > and N OMF files, and run a script.
> 
> If I understand well, "install" for docs and OMF files means only
> copying, and therefore it's okay.
> 
> The problems start to arise when you try using a tool that modifies a
> database. You will have to add at least a %Post section and a %Postun
> section. I can foresee distributions saying "This OMF thingie is nice,
> but it's too much work for our packagers, and it will slow down even
> more our post-installation phase. Let's forget about it." You can assume
> that developpers will like it and at lines to heir "make install". But I
> wouldn't bet on packagers' enthusiasm. Maybe I'm too pessimistic, but...
> 
> >  The docs and OMF files which are
> > installed are used by scrollkeeper but not modified after
> > installation.  They serve as reference for scrollkeeper to build its
> > database from, but do not directly comprise the database.  At install
> > (ie. %Post), scrollkeeper clearly needs to use these two files to add info
> > to its database files.  After this time, they aren't used
> > that much by scrollkeeper.  I think we will probably want something
> > analogous to the --rebuilddb feature to RPM which will allow scrollkeeper
> > to use its simple list of OMF files on the system and the OMF files
> > themselves to rebuild all of the database files.  Presumably this is only
> > necessary if the database files are accidentally harmed ("darn - I guess I
> > shouldn't have hit ctrl-C during an install").  The original files will
> > probably be used sometimes when when scrollkeeper is upgraded to extract
> > new information which wasn't needed in previous databases but is for the
> > new database. (eg. when new versions of scrollkeeper have new
> > functionality)
> 
> The only opportunity to avoid having a %Post and a %Postun are to ask
> for a manual "--rebuilddb" (ush - everyone will forget that when he/she
> installs or uninstalls software) or to have it rebuilt automatically
> when some caching mechanism detects that the database is too old.
> 
> > > This was misunderstanding from me, I thought the OMF files were
> > > concatenated. Forget about the OMF files. If they are just copied "as
> > > is" then it's perfect.
> >
> > Yes.  We want to copy them "as is", but then use them with a postinstall
> > script which modifies the real database files under /var.
> 
> This is the problematic part.
> 
> > I think we want something along the lines of (b). 'make install:' installs
> > the doc(s) and omf file(s). Then it calls scrollkeeper-install, pointing
> > it to the doc(s) and omf file(s) it installed and giving it the path
> > $prefix/var/scrollkeeper.  If it is an RPM install, this path is something
> > like /tmp/scrollkeeper-0.1/var/scrollkeeper.
> 
> Just a technical detail: not really. There must be provisions in the
> software to make transparent the fact that during build it was in /tmp.
> 
> >  If it is a real install it
> > is /var/scrollkeeper.  We have a couple choices on how
> > scrollkeeper-install works.  I think the easiest is that if the database
> > doesn't exist, scrollkeeper just exits.
> 
> It will never exist before: remember, you are in tmp/.
> 
> >  In this case for a person
> > building an RPM, scrollkeeper-install looks for its main list,
> > /tmp/scrollkeeper-0.1/var/scrollkeeper/scrollkeeper-docs.  This file
> > doesn't exist, so scrollkeeper-install just exits.  If 'make install:' is
> > run by hand, scrollkeeper-install wants to append an entry to
> > /var/scrollkeeper/scrollkeeper-docs which exists, so scrollkeeper-install
> > proceeds happily updating the various database files in /var/scrollkeeper.
> > We will need $POST to have scrollkeeper-install.
> >
> > Another option would be to generate fresh database files under
> > /tmp/scrollkeeper-0.1/var/scrollkeeper if they don't exist already, but I
> > really don't see any point in doing this since they won't be installed and
> > they will be deleted a couple seconds later.
> 
> No, it would be the "usual" way of doing it: the freshly created
> database holds the partial (= only for this package) information and
> it's this partial database that is merged at postinstallation's time on
> the end-user's hard disk. Or even better, this partial database is only
> copied but then we have to cope with a database split into as many files
> as there are packages with OMF information.
> 
> > > Even if you could, it would be a bad idea. A "make install" instruction
> > > is supposed to be run equally from the end user's command line and from
> > > a packager's spec file.
> >
> > I'm not sure exactly what this means.  Does my paragraph above satisfy
> > this requirement?  If not, what would need to be changed?  Certainly it
> > would not be required that the real databases are changed.
> 
> Well, as far as scrollkeeper exits without producing an error code
> different than 0 when trying to "make install" if the database does not
> exist previously, then yes, you can assume that it can do whatever it
> wants, either not create a fresh one or create it. But it must also
> behave the same way on the user's hard disk if the database did not
> exist previously.
> 
> "make install" must behave the same way on the user's hard disk and on
> the packager's hard disk. If the database does not exist previously and
> if it exits then with an error code, then we have a problem.
> 
> --
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel