Re: [Scrollkeeper-devel] What about the distributions?

[Dan M. <d-...@uc...>]

On Fri, 27 Oct 2000, Eric Bischoff wrote:

> Le Mercredi 25 Octobre 2000 17:57, vous avez écrit :
> > 1) When and how does the cache file (ie. big database) get created from
> > the smaller ones?  I think the previous suggestion was that whenever
> > scrollkeeper is queried for information, it first makes sure all the small
> > databases have been merged into the big cache database.
> 
> According to me, it's the best solution.
> 
> It unfortunately means that the first time doc is searched, all the small 
> files built during installation distribution are merged and it can last a 
> while. But it's no worst than the post-installation phase. 

It is worse, because an admin is willing to wait five seconds or more for
a package to install, but users don't want to wait 5 extra seconds when
they open their help browser.

> >  One big problem I
> > see with this is permissions and security.  If a user running
> > foo-help-browser asks scrollkeeper for the contents list and scrollkeeper
> > realizes it needs to update its big cache database from the small ones, it
> > will need to have permission to write to the scrollkeeper files.  This
> > means we run suid root programs,
> 
> Urgh
> 
> > or create a special scrollkeeper user
> > which runs it.  The former seems like a security problem and could cause
> > problems with getting people to use and distribute scrollkeeper.
> 
> Very true.
> 
> I might say something stupid, but why can't the contents list be predefined? 
> Or is it imperative to build it out of OMF data?

I'm not sure I follow your question.  In order to build the contents list,
you have to know which packages are installed on the computer.  There is
no way of knowing this until you are on the user's machine.  Further, it
would be very nice if the categorization list of the contents list was on
the user's machine.  This way an admin could choose not to have certain
categories in the contents list.  For example if I was an admin for a set
of machines which I knew newbies were using, I might chop things like man
pages or LDP HOWTOs out of the Contents List since they would only confuse
newbies further.  I might also want to add my own categories for fields of
interest for my users.

> Another solution, if the contents list is dynamic but small, is to create it 
> in the user's home directory. It's a bit of redundant work to scrollkeeper to 
> do this for each user, but it also enables each user to build a contents list
> that matches the best its inline searches.

I am hesitant to do this, partly because it seems strange and partly
because it will cause a performance problem. At the same time, I like the
idea of having this as an option for the future since it would potentially
allow each user to configure their own Contents List, which would be
great.  I think the default behavior should not be to replicate itself
into each user's $HOME directory, but to use the system-wide database
unless a user explicitly wants to add or remove contents.  This would be
feature which we would not implement for a long time anyway.  If and when
we implement it, we can probably get around the performance problem by
copying the system-wide database to start with.  However not having a
system-wide database means that each user gets completely hammered the
first time they open their browser.

> >  The
> > latter would work, but is a big pain since it means creating new entries
> > in /etc/password, /etc/shadow, /etc/groups, etc.  I don't even want to
> > think what will happen on NIS networks.
> 
> ;-)
> 
> >  This seems to have its own set of
> > security issues and nasty complexities.  I think this would force us to
> > have each user have their own scrollkeeper database under
> > $HOME/.scrollkeeper/.
> 
> Hey we have our brains wired the same way ;-)

:)

> > 2) Note that it also means that the first time to start up your
> > foo-help-browser after installing a few packages, you get a quite
> > noticable performance penalty.
> 
> Wow the same thoughts ;-)
> 
> > 3) It also does not provide feedback to the administrator.  Suppose the
> > sys admin installs foo which has a corrupt scrollkeeper database, or just
> > one which does not match the version of scrollkeeper on the machine.  It
> > installs fine, since it just copies the file.  The admin leaves, a user
> > comes along and runs the help browser, scrollkeeper tries to merge the
> > database... if things break, the sys admin is gone.  I think the whole
> > installation process should be done at install time, not half at install
> > time by the admin and half at run-time by the user.
> 
> But at install time it means postinstall scripts. It's technically better, 
> but I can't figure out how distributions will react. It was painful enough to 
> change to LSB directory tree at Caldera.

I agree this is a very important issue.  We have to be confident the
distributions will accept this.  However changing the directory tree is a
*much* bigger concern than adding scrollkeeper.  Plus, scrollkeeper can be
added in little pieces.  Scrollkeeper will be adopted slowly, giving
distributions time to get familiar with it.  If GNOME 1.4 can use it, as I
am desperately hoping, then that means that maybe 20 packages will be
using it in the next few months.  (GNOME 1.4 should be out in about 6
weeks, if it can stay on schedule ;)  If KDE adopts it for their next
release along with a few other people, we are looking at maybe 70 packages
5 or 6 months from now.  That isn't so much to really swamp anybody.  
Plus, I'm planning that the lines we add to the spec files are completely
generic.  Basically just cut and paste 4 lines into the spec file and you
are done.  If you forget, then that doc doesn't show up in the contents
list. It doesn't really break anything, so I don't think anybody is going
to scream about a distribution forgetting to add it to a couple spec
files.

I think it would help a lot to actually try to implement this
solution.  Perhaps it will turn out to be more complicated than I am
thinking?

> > 4) Versioning becomes a very big problem.  Suppose a person is running
> > scrollkeeper 0.2.  If he installs a mini-db from a package where the build
> > machine was running scrollkeeper-0.1 or scrollkeeper-0.3, the mini-db will
> > potentially have the wrong format.  The best case scenario is scrollkeeper
> > ignores the file as if it didn't exist.  The worst case scenario is that
> > things break or get corrupted.
> 
> This problem you will get anyway with OMF files.
> 
> The solution is the database format used in the database. If it's 
> XML-conformant then it's much easier to have different XML formats
> in the same database. XML allows backward and forward compatibility if your 
> changes respect very minimalistic rules. For example you can add and remove 
> tags.

XML is great, I agree.  However in this case there is a much better
solution than just having XML give us the ability to not have information.  
If the user upgrades from scrollkeeper-0.1 to scrollkeeper-0.2 (or has
scrollkeeper-0.2 and downloads a package built with scrollkeeper-0.1),
they could keep their old XML mini-databases, scrollkeeper can be made not
to crash, and the new features in scrollkeeper-0.2 just won't work on the
older mini-databases. So the new features will not really be very usable
for 6 months or so until most packages have new releases.  However it is a
huge shame to use a solution like this when we know that all the
information we need is sitting in the original SGML and OMF files on the
hard drive and scrollkeeper knows exactly where they are at.  If I were
the scrollkeeper, I would say "Forget this obsolete mini-db, I know where
the information is at and I will go extract it and merge it into my real
database":)

> > I am omitting one possibility - scrollkeeper upgrades old mini-dbs to the
> > new format.  I don't think we can do this because (1) the files it would
> > be modifying "belong" to the package foo, not to scrollkeeper, and (2) it
> > would not really be "updating" them. Instead it would have to go to the
> > original OMF file since the information it needs is not necessarily in the
> > old mini-db.  In this case, it is completely obsoleting the mini-db.  The
> > OMF file and the doc have all the information needed.  The mini-db just
> > has a subset of the information needed.  Plus it would not know how to
> > "downgrade" a mini-db since scrollkeeper-0.2 doesn't know what changed
> > between scrollkeeper-0.2 and scrollkeeper-0.3.
> >
> > Generally, this means that a person cannot upgrade any packages on his
> > system (scrollkeeper or foo) away from original distributions packages
> > unless he can be sure the packager was using the same version of
> > scrollkeeper as the user has.  I think most Linux users would be very
> > upset by this.  (One of the best things about Linux is I can go out and
> > get the latest version of scrollkeeper or foo every month or two. Losing
> > this ability would be a big problem.)  Generally, it leads to even more
> > specificity of RPMs.  Right now, I can typically download an RPM for RH
> > 6.0 or 6.1 or Mandrake and sometimes some others and they will run fine on
> > RH6.2.  Having the databases generated on the build machine will almost
> > guarantee to break this.  So the user is given a smaller set of packages
> > he can install and he is pressured into sticking with his distribution for
> > getting updates.
> >
> > 5) In most cases, it will be easier to add data into the single large
> > cache database by going directly to the OMF file and the doc than by
> > trying to use a mini-db.
> 
> Again, post-install scripts *are* the technical solution, but will upset the 
> distributions. There's no need for you to prove that they are better, we 
> already know it.

My talks with Jonathan and Jeff (via. Jonathan) at Red Hat have indicated
that they did not feel this would place an undo burden on distributors.  
That is my impression as well, although you (Eric), Jonathan, and Jeff
have much more expertise in this area than I do.

> >  In most cases, it is just a matter of writing
> > one set of parsing/writing functions instead of two.  (It is not a simple
> > concatenation of two files.)  In some cases, I'm not sure if it will even
> > be possible to "merge" databases unless we have a separate intermediate
> > format to the database.  This is basically because I can give instructions
> > to Bill on how to cut a piece of XML out of a file but Bob will not
> > necessarily know where to put the XML back in later unless I give Bob
> > additional information.
> >
> > Sorry for the big list of problems, but I feel these are valid concerns.
> 
> Yes. I think we are in the heart of the problem. Sorry to have started all of 
> this, but it was a mere fact than in a distribution a "make install" is done 
> by the packager, not by the end user.

It is great to have your help.  My original plan would have had many OMF
files in each spec file and many calls to post-install scripts, which
would have made quite a bit of work for packagers.

The difference in configuration between the packager's machine and the
end-user's machine is a big part of why I prefer the post-install script
solution.

> > Given my current understanding of the mini-database idea, I feel the
> > single database is a better solution.
> 
> It is. No discussion. If we had a way of not having the packagers explicitely 
> write the calls to those postinstall and postuninstall scripts it would be 
> perfect.

I am hoping that we can make this process very simple for packagers (as
well as for developers).  I am confident that we can make packagers' lives
prety simple here while still using post-install scripts.  We haven't
discussed the developers' job on this list yet, although I have done so
somewhat with Laszlo in private.  I'll bring up the one issue which sticks
out as tricky right now:

Suppose we have a package with two apps, foo1 and foo2, each with a
doc.  So we have:
foo1/docs/foo1-manual.xml
foo1/docs/foo1-manual.omf
foo2/docs/foo2-manual.xml
foo2/docs/foo2-manual.omf

Now we want to use the pre-install script as you suggested.  This script
should basically do two things:
1) Allow us to set the URL in the OMF file to point to where the doc will
be installed. This is relatively straightforward, so I won't talk about it
any more right now.
2) Merge the OMF files into one big OMF file.

While #2 seems relatively trivial (we could almost do a "cat <filename> >>
<bigfilename>" where <bigfilename> will probably need to be the relative
path to the file with some "../.." in it), I'm not sure how to get it to
work right with the build process and autoconf.  For simplicity, imagine
we use 'cat' as above.  We would like to make sure that if somebody does
'make' twice in a row, it does not just keep appending the same entries
into <bigfilename>, duplicating them.  It would seem that we want to
delete <bigfilename> each time we start a 'make', but I don't see how we
can do this with autoconf/automake.  (Anybody know?)  Another solutions
would be to have scrollkeeper-preinstall be smart enough to know if it is
duplicating an entry and just update instead of add an all new entry.  
This would not delete an entry if one was removed.  Another option would
be to assume that people always do 'make clean' between makes, but I
imagine that builders and developers monkey around a fair amount without
doing a 'make clean' so this may be a bad assumption.  Any ideas for a
solution to this one?  Remember - it has to be absolutely trivial for
developers and packagers.

> >  The single database solution has
> > the extra complexity of having a post-install script, but I don't think
> > adding a couple trivial lines to a spec file is that bad.  I think it is
> 
> A post-install script is not something complicated. It is generating a 
> problem with the human factor. Those lines have to be added in about a 
> thousand spec files in each distribution and the documentation is not usually 
> the top priority anywhere.

Well, let's say about 50 or 100 packages maximum in the next year or
so.  I think the combination of making it very easy and the fact that an
infrequent ommission will not have very serious consequences will help
too.

> Let me state the problem the other way round. We won't merge the OMF nor the 
> DocBook files either. So why should we merge the extra data and not these 
> ones?

Aside from all the reasons I've already stated? ;)

The docs and OMF files belong to the individual packages to install and
uninstall as necessary to remain in sync with the packages.  The
scrollkeeper database belongs to scrollkeeper, to maintain, update,
synchronize, modify, and use as scrollkeeper needs.  I think having
scrollkeeper's database created, shipped, and maintained (or in this case
*not maintained*) by the applications (in many pieces) instead of
by scrollkeeper greatly restricts scrollkeeper's ability to do its job.

> > much simpler than the problems we have to address above and provides for
> > a more robust, secure, and flexible end-product.
> 
> Sure.
> 
> > BTW: There is a somewhat intermediate solution which involves having each
> > package foo add itself to a queue which should be processed by
> > scrollkeeper.
> 
> And how does it add itself to the queue? In the post-install script? ;-)
> 
> This adds nothing new to the choice between post-install and mini-databases.
> 
> > It moves the database work onto the end-user machine, like
> > the 1-database solution. This solves the versioning problems discussed
> > above.  But it (optionally) gets rid of the post-install script.  This
> > could make the RPMs simpler (minus 4 lines), but leaves the security and
> > performance problems mentioned above. It also eliminates the extra coding
> > work needed for the extra conversion step in OMF/dock->min-db->db.  This
> > is very close to my original proposal to the Red Hat guys which they did
> > not seem to care for.
> 
> I'm not sure I understand what you propose here.

Well, I prefer the post-install script, which is why I do not emphasize
this solution.  However, if we *really* wanted to avoid the script, we
could do something like: Have a magic directory every package installs a
file into, say /var/lib/scrollkeeper/packages.  Each package installs a
short file in here which lists the full path to the OMF file(s) installed
by the package.  Then whenever scrollkeeper is run it checks to see if
there are any new, modified, or missing files in this directory.  If so,
it updates its database.  The advantage is that it still uses one database
making things simpler and it can keep the versions correct.  The downside
is that it still has the permissions and performance problem, it requires
all packages to install something into a special directory, and it is
generally hokey.  Plus it has to detect when a file has been modified,
which means either comparing the contents or checking timestamps, neither
of which seem pretty.

Dan