Re: [Scrollkeeper-devel] What about the distributions?

[Dan M. <d-...@uc...>]

Hi everybody,

First - I'm sorry I dropped off this list for a few days.  I have been
very busy.  The time I have been putting into ScrollKeeper was confined to
IRC, discussing these problems and possible solutions with Laszlo,
Jonathan, one of Red Hat's main packagers, and a couple other people.

I think we have resolved all the issues to having a main database (as
opposed to the mini-database idea, which I discuss below).  As Eric
suggests, we should merge the OMF files together in a pre-install step.  
This keeps things simple for the packager, since it is just 1 extra files
instead of N.  As part of 'make install', it adds the new information to
the big database.  For RPM *building*, it will be under BUILDROOT, which
means it creates a fresh database under BUILDROOT. In this "one big
database" approach, we do not keep the fresh database created under
BUILDROOT and install it.  We just have it get built to make sure no
errors occur, then it will be deleted.  Plus we can go snoop around in
there if we want to do testing.  The final package just gets the one new
big-OMF file.  It also gets a new line for %post and a new line for
%uninstall.

The other option is to have many mini-database files.  Each package
creates and installs a mini-database into a standard location, say
/var/lib/scrollkeeper/mini-dbs/ for the sake of discussion.  There are
many variations on how we can proceed from there, but I think the main
idea is to get rid of %post and %uninstall altogether.  So lets assume the
installation just installs the mini-db file.  As we discussed previously
on this list, merging the database files will take a certain amount of
time.  I'm guessing something like a second per package very roughly.  
(It has to read and parse the OMF file and all the docs in the package.)  
So to do this for 1000 packages would take a long time for a user to wait
to open his browser.  So, we would have to create cache files which are
incrementally added too (ie. the one big database above).  Now we have a
number of options on how we do things, but generally I see the following
problems:

1) When and how does the cache file (ie. big database) get created from
the smaller ones?  I think the previous suggestion was that whenever
scrollkeeper is queried for information, it first makes sure all the small
databases have been merged into the big cache database.  One big problem I
see with this is permissions and security.  If a user running
foo-help-browser asks scrollkeeper for the contents list and scrollkeeper
realizes it needs to update its big cache database from the small ones, it
will need to have permission to write to the scrollkeeper files.  This
means we run suid root programs, or create a special scrollkeeper user
which runs it.  The former seems like a security problem and could cause
problems with getting people to use and distribute scrollkeeper.  The
latter would work, but is a big pain since it means creating new entries
in /etc/password, /etc/shadow, /etc/groups, etc.  I don't even want to
think what will happen on NIS networks.  This seems to have its own set of
security issues and nasty complexities.  I think this would force us to
have each user have their own scrollkeeper database under
$HOME/.scrollkeeper/.  I think we will still have a huge time delay the
first time a user starts up their help browser.

2) Note that it also means that the first time to start up your
foo-help-browser after installing a few packages, you get a quite
noticable performance penalty.

3) It also does not provide feedback to the administrator.  Suppose the
sys admin installs foo which has a corrupt scrollkeeper database, or just
one which does not match the version of scrollkeeper on the machine.  It
installs fine, since it just copies the file.  The admin leaves, a user
comes along and runs the help browser, scrollkeeper tries to merge the
database... if things break, the sys admin is gone.  I think the whole
installation process should be done at install time, not half at install
time by the admin and half at run-time by the user.

4) Versioning becomes a very big problem.  Suppose a person is running
scrollkeeper 0.2.  If he installs a mini-db from a package where the build
machine was running scrollkeeper-0.1 or scrollkeeper-0.3, the mini-db will
potentially have the wrong format.  The best case scenario is scrollkeeper
ignores the file as if it didn't exist.  The worst case scenario is that
things break or get corrupted.

I am omitting one possibility - scrollkeeper upgrades old mini-dbs to the
new format.  I don't think we can do this because (1) the files it would
be modifying "belong" to the package foo, not to scrollkeeper, and (2) it
would not really be "updating" them. Instead it would have to go to the
original OMF file since the information it needs is not necessarily in the
old mini-db.  In this case, it is completely obsoleting the mini-db.  The
OMF file and the doc have all the information needed.  The mini-db just
has a subset of the information needed.  Plus it would not know how to
"downgrade" a mini-db since scrollkeeper-0.2 doesn't know what changed
between scrollkeeper-0.2 and scrollkeeper-0.3.

Generally, this means that a person cannot upgrade any packages on his
system (scrollkeeper or foo) away from original distributions packages
unless he can be sure the packager was using the same version of
scrollkeeper as the user has.  I think most Linux users would be very
upset by this.  (One of the best things about Linux is I can go out and
get the latest version of scrollkeeper or foo every month or two. Losing
this ability would be a big problem.)  Generally, it leads to even more
specificity of RPMs.  Right now, I can typically download an RPM for RH
6.0 or 6.1 or Mandrake and sometimes some others and they will run fine on
RH6.2.  Having the databases generated on the build machine will almost
guarantee to break this.  So the user is given a smaller set of packages
he can install and he is pressured into sticking with his distribution for
getting updates.

5) In most cases, it will be easier to add data into the single large
cache database by going directly to the OMF file and the doc than by
trying to use a mini-db.  In most cases, it is just a matter of writing
one set of parsing/writing functions instead of two.  (It is not a simple
concatenation of two files.)  In some cases, I'm not sure if it will even
be possible to "merge" databases unless we have a separate intermediate
format to the database.  This is basically because I can give instructions
to Bill on how to cut a piece of XML out of a file but Bob will not
necessarily know where to put the XML back in later unless I give Bob
additional information.

Sorry for the big list of problems, but I feel these are valid concerns.
Given my current understanding of the mini-database idea, I feel the
single database is a better solution.  The single database solution has
the extra complexity of having a post-install script, but I don't think
adding a couple trivial lines to a spec file is that bad.  I think it is
much simpler than the problems we have to address above and provides for
a more robust, secure, and flexible end-product.

Dan

BTW: There is a somewhat intermediate solution which involves having each
package foo add itself to a queue which should be processed by
scrollkeeper. It moves the database work onto the end-user machine, like
the 1-database solution. This solves the versioning problems discussed
above.  But it (optionally) gets rid of the post-install script.  This
could make the RPMs simpler (minus 4 lines), but leaves the security and
performance problems mentioned above. It also eliminates the extra coding
work needed for the extra conversion step in OMF/dock->min-db->db.  This
is very close to my original proposal to the Red Hat guys which they did
not seem to care for.