scrollkeeper-devel

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Mon, 23 Oct 2000, laszlo kovacs wrote:

> > Yes, but when the RPM is "done building" it cleans up after itself so the file
> > would no longer exist under that location.
> > 
> > Perhaps what we need are two directories or something. So you would have:
> > prefix/var/lib/scrollkeeper/processed
> > prefix/var/lib/scrollkeeper/unprocessed
> > 
> > An "initial" scrollkeeper-install would put the file in the unprocessed
> > section, scrollkeeper could then at some designated time, check all files in
> > the unprocessed section, if they are found on the local drive they are moved
> > to processed, otherwise they are removed from unprocessed.
> > 
> > But this presents another issue. What about "online docs"? the above
> > implementation would force people to have docs on the "local" path!!! Instead
> > of deleting the file, perhaps we should move it into a "not-installed"
> > directory. That way if the help file is located at
> > "http://www.doc.com/file.sgml" We could access it from the not-installed
> > section.
> > 
> > The problem would arise with "syncing" the two directories (what if you have a
> > "not-installed" doc and then later install it locally)
> > 
> > So perhaps we shouldn't have multiple databases (or directories, or whatever
> > you want to call it). Perhaps we should always install files...
> > 
> > If a doc is not found/installed locally then ScrollKeeper should handle it.
> > This sounds a lot like what Eric was saying doesn't it? ;) But he makes it
> > sound that applications (help browsers) should handle it. Instead, I think
> > that ScrollKeeper should natively handle it.
> > 
> > If I ask the ScrollKeeper database for "all installed docs" it would return
> > that - I /could/ ask for all "not-installed" docs (which would return these
> > docs + the HTTP ones + FTP ones + etc...)
> > 
> > Generally, when a help browser displays its "Index" (for Nautilus that is the
> > Help Sidebar) it would only display the "installed" docs...
> > 
> > What do you think of this proposal?
> > 
> > Regards,
> > Ali
> 
> I dont think I managed to understand this proposal. Does it adress the
> problem of post install scripts? We can install docs to a unified
> Scrollkeeper database only through a post install script or let the help
> browser to deal with the individual metadata of each package separately.
> Is this adressed in the proposal above?

I am saying USE a post-install script all the time whenever a 'make 
install' is done. ScrollKeeper would then be (internally) intelligent 
enough and able to distinguish an "installed" file from an "uninstalled" 
file. (the "uninstalled" (better name is "non-local") files are files in 
the DB, but not on the HD, and any help files on the internet (HTT)

Does this make it clearer?
 
Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> I am saying USE a post-install script all the time whenever a 'make
> install' is done. ScrollKeeper would then be (internally) intelligent
> enough and able to distinguish an "installed" file from an "uninstalled"
> file. (the "uninstalled" (better name is "non-local") files are files in
> the DB, but not on the HD, and any help files on the internet (HTT)
> 
> Does this make it clearer?

????

Your proposal enables to get rid of the post installs and what was nice
in it.


-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Dan Mueth wrote:
> 
> I don't think it is at all possible, or even desireable, to have each
> portion of the database broken into 1000 files.

... then it means that you will force the update of 1000 SPEC files in
each distribution :-(.

>  We will have a handful of
> database file types, so this would mean something like an extra 5000 files
> installed.  It would largely defeat the purpose of having the database
> generated in advance, since scrollkeeper would have to figure out how to
> piece these together every time it is run by the help browser.  For
> example, if the help browser wants the contents list (which is sorted by
> subject), scrollkeeper would have to read all 1000 files, look at the
> contents list categorization description, and then sort and merge all the
> entries.  This would be a big performance problem - every time you open
> your help browser it has to do this over.

... unless there is a caching mechanism that rebuilds the list only when
it is needed (for example each time the number of files in the database
directory has changed). Not easy though.

>  Further, if we ever change the
> format of any of these files at all, having all the work done on the
> builder's machine will guarantee that everybody's system will have file
> version incompatibilities (some packages were built with the old
> scrollkeeper).

This is a generic problem with distributions. I'm facing it for example
with the docbook-tools. A file built on some system often has problems
to run on another system. Basically people assume that once you have a
distribution installed on your machine then you must download any
further software on the same distribution's sites.

What you say here is independant from having 1 or 1000 files in the
database: whether you like it or not, "make install" will be run on the
packager's machine, not on the user's machine. You have to be very
conscious of this.

>  If it is all done on the users machine, then when you
> upgrade scrollkeeper from version 0.X to 0.X+1, a script can run to
> migrate the database if necessary between the old format to the new one.
> I think we are almost guaranteed to want to do this.  While it would be
> nice to design our database files perfectly in our first shot, I really
> don't expect this to happen.

You cannot assume that "make install" is run on the end user's machine.
I think you are starting to see the consequences of this simple fact
only now ;-).

> In fact, we even have problems with adding features.  Suppose
> scrollkeeper-0.2 has a new feature, say searching, with a new database.
> When you upgrade to version 0.2, scrollkeeper should have an install
> script which generates the new database using its list of all the OMF
> files and docs on the system.  Suddenly, the user now has searching
> ability with a complete database :)  If we left all of this work up to the
> build machines, the users would have to wait many months until most
> builders decide to upgrade their scrollkeeper to 0.2 and rebuild the RPMs
> for you.  Then you have to get new versions of each of these 1000
> packages before you can use the new feature (searching).

This is a problem of updates any tool has with distributions.
- either the user gathers his own distribution himself and compiles and
"make install" himself
- either he becomes dependant of the quality of the work of the
packager. Nothing prevents a distribution from using outdated tools.

> Basically, we want each application to install N docs, install between 1
> and N OMF files, and run a script.

If I understand well, "install" for docs and OMF files means only
copying, and therefore it's okay.

The problems start to arise when you try using a tool that modifies a
database. You will have to add at least a %Post section and a %Postun
section. I can foresee distributions saying "This OMF thingie is nice,
but it's too much work for our packagers, and it will slow down even
more our post-installation phase. Let's forget about it." You can assume
that developpers will like it and at lines to heir "make install". But I
wouldn't bet on packagers' enthusiasm. Maybe I'm too pessimistic, but...

>  The docs and OMF files which are
> installed are used by scrollkeeper but not modified after
> installation.  They serve as reference for scrollkeeper to build its
> database from, but do not directly comprise the database.  At install
> (ie. %Post), scrollkeeper clearly needs to use these two files to add info
> to its database files.  After this time, they aren't used
> that much by scrollkeeper.  I think we will probably want something
> analogous to the --rebuilddb feature to RPM which will allow scrollkeeper
> to use its simple list of OMF files on the system and the OMF files
> themselves to rebuild all of the database files.  Presumably this is only
> necessary if the database files are accidentally harmed ("darn - I guess I
> shouldn't have hit ctrl-C during an install").  The original files will
> probably be used sometimes when when scrollkeeper is upgraded to extract
> new information which wasn't needed in previous databases but is for the
> new database. (eg. when new versions of scrollkeeper have new
> functionality)

The only opportunity to avoid having a %Post and a %Postun are to ask
for a manual "--rebuilddb" (ush - everyone will forget that when he/she
installs or uninstalls software) or to have it rebuilt automatically
when some caching mechanism detects that the database is too old.

> > This was misunderstanding from me, I thought the OMF files were
> > concatenated. Forget about the OMF files. If they are just copied "as
> > is" then it's perfect.
> 
> Yes.  We want to copy them "as is", but then use them with a postinstall
> script which modifies the real database files under /var.

This is the problematic part.

> I think we want something along the lines of (b). 'make install:' installs
> the doc(s) and omf file(s). Then it calls scrollkeeper-install, pointing
> it to the doc(s) and omf file(s) it installed and giving it the path
> $prefix/var/scrollkeeper.  If it is an RPM install, this path is something
> like /tmp/scrollkeeper-0.1/var/scrollkeeper.

Just a technical detail: not really. There must be provisions in the
software to make transparent the fact that during build it was in /tmp.

>  If it is a real install it
> is /var/scrollkeeper.  We have a couple choices on how
> scrollkeeper-install works.  I think the easiest is that if the database
> doesn't exist, scrollkeeper just exits.

It will never exist before: remember, you are in tmp/.

>  In this case for a person
> building an RPM, scrollkeeper-install looks for its main list,
> /tmp/scrollkeeper-0.1/var/scrollkeeper/scrollkeeper-docs.  This file
> doesn't exist, so scrollkeeper-install just exits.  If 'make install:' is
> run by hand, scrollkeeper-install wants to append an entry to
> /var/scrollkeeper/scrollkeeper-docs which exists, so scrollkeeper-install
> proceeds happily updating the various database files in /var/scrollkeeper.
> We will need $POST to have scrollkeeper-install.
> 
> Another option would be to generate fresh database files under
> /tmp/scrollkeeper-0.1/var/scrollkeeper if they don't exist already, but I
> really don't see any point in doing this since they won't be installed and
> they will be deleted a couple seconds later.

No, it would be the "usual" way of doing it: the freshly created
database holds the partial (= only for this package) information and
it's this partial database that is merged at postinstallation's time on
the end-user's hard disk. Or even better, this partial database is only
copied but then we have to cope with a database split into as many files
as there are packages with OMF information.

> > Even if you could, it would be a bad idea. A "make install" instruction
> > is supposed to be run equally from the end user's command line and from
> > a packager's spec file.
> 
> I'm not sure exactly what this means.  Does my paragraph above satisfy
> this requirement?  If not, what would need to be changed?  Certainly it
> would not be required that the real databases are changed.

Well, as far as scrollkeeper exits without producing an error code
different than 0 when trying to "make install" if the database does not
exist previously, then yes, you can assume that it can do whatever it
wants, either not create a fresh one or create it. But it must also
behave the same way on the user's hard disk if the database did not
exist previously.

"make install" must behave the same way on the user's hard disk and on
the packager's hard disk. If the database does not exist previously and
if it exits then with an error code, then we have a problem.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Laszlo K. <las...@su...>]

I dont seem to be able to find the mail you replied to in this one. Was
that between Dan and yourself only?

Laszlo

> 
> Dan Mueth wrote:
> >
> > I don't think it is at all possible, or even desireable, to have each
> > portion of the database broken into 1000 files.
> 
> ... then it means that you will force the update of 1000 SPEC files in
> each distribution :-(.
> 
> >  We will have a handful of
> > database file types, so this would mean something like an extra 5000 files
> > installed.  It would largely defeat the purpose of having the database
> > generated in advance, since scrollkeeper would have to figure out how to
> > piece these together every time it is run by the help browser.  For
> > example, if the help browser wants the contents list (which is sorted by
> > subject), scrollkeeper would have to read all 1000 files, look at the
> > contents list categorization description, and then sort and merge all the
> > entries.  This would be a big performance problem - every time you open
> > your help browser it has to do this over.
> 
> ... unless there is a caching mechanism that rebuilds the list only when
> it is needed (for example each time the number of files in the database
> directory has changed). Not easy though.
> 
> >  Further, if we ever change the
> > format of any of these files at all, having all the work done on the
> > builder's machine will guarantee that everybody's system will have file
> > version incompatibilities (some packages were built with the old
> > scrollkeeper).
> 
> This is a generic problem with distributions. I'm facing it for example
> with the docbook-tools. A file built on some system often has problems
> to run on another system. Basically people assume that once you have a
> distribution installed on your machine then you must download any
> further software on the same distribution's sites.
> 
> What you say here is independant from having 1 or 1000 files in the
> database: whether you like it or not, "make install" will be run on the
> packager's machine, not on the user's machine. You have to be very
> conscious of this.
> 
> >  If it is all done on the users machine, then when you
> > upgrade scrollkeeper from version 0.X to 0.X+1, a script can run to
> > migrate the database if necessary between the old format to the new one.
> > I think we are almost guaranteed to want to do this.  While it would be
> > nice to design our database files perfectly in our first shot, I really
> > don't expect this to happen.
> 
> You cannot assume that "make install" is run on the end user's machine.
> I think you are starting to see the consequences of this simple fact
> only now ;-).
> 
> > In fact, we even have problems with adding features.  Suppose
> > scrollkeeper-0.2 has a new feature, say searching, with a new database.
> > When you upgrade to version 0.2, scrollkeeper should have an install
> > script which generates the new database using its list of all the OMF
> > files and docs on the system.  Suddenly, the user now has searching
> > ability with a complete database :)  If we left all of this work up to the
> > build machines, the users would have to wait many months until most
> > builders decide to upgrade their scrollkeeper to 0.2 and rebuild the RPMs
> > for you.  Then you have to get new versions of each of these 1000
> > packages before you can use the new feature (searching).
> 
> This is a problem of updates any tool has with distributions.
> - either the user gathers his own distribution himself and compiles and
> "make install" himself
> - either he becomes dependant of the quality of the work of the
> packager. Nothing prevents a distribution from using outdated tools.
> 
> > Basically, we want each application to install N docs, install between 1
> > and N OMF files, and run a script.
> 
> If I understand well, "install" for docs and OMF files means only
> copying, and therefore it's okay.
> 
> The problems start to arise when you try using a tool that modifies a
> database. You will have to add at least a %Post section and a %Postun
> section. I can foresee distributions saying "This OMF thingie is nice,
> but it's too much work for our packagers, and it will slow down even
> more our post-installation phase. Let's forget about it." You can assume
> that developpers will like it and at lines to heir "make install". But I
> wouldn't bet on packagers' enthusiasm. Maybe I'm too pessimistic, but...
> 
> >  The docs and OMF files which are
> > installed are used by scrollkeeper but not modified after
> > installation.  They serve as reference for scrollkeeper to build its
> > database from, but do not directly comprise the database.  At install
> > (ie. %Post), scrollkeeper clearly needs to use these two files to add info
> > to its database files.  After this time, they aren't used
> > that much by scrollkeeper.  I think we will probably want something
> > analogous to the --rebuilddb feature to RPM which will allow scrollkeeper
> > to use its simple list of OMF files on the system and the OMF files
> > themselves to rebuild all of the database files.  Presumably this is only
> > necessary if the database files are accidentally harmed ("darn - I guess I
> > shouldn't have hit ctrl-C during an install").  The original files will
> > probably be used sometimes when when scrollkeeper is upgraded to extract
> > new information which wasn't needed in previous databases but is for the
> > new database. (eg. when new versions of scrollkeeper have new
> > functionality)
> 
> The only opportunity to avoid having a %Post and a %Postun are to ask
> for a manual "--rebuilddb" (ush - everyone will forget that when he/she
> installs or uninstalls software) or to have it rebuilt automatically
> when some caching mechanism detects that the database is too old.
> 
> > > This was misunderstanding from me, I thought the OMF files were
> > > concatenated. Forget about the OMF files. If they are just copied "as
> > > is" then it's perfect.
> >
> > Yes.  We want to copy them "as is", but then use them with a postinstall
> > script which modifies the real database files under /var.
> 
> This is the problematic part.
> 
> > I think we want something along the lines of (b). 'make install:' installs
> > the doc(s) and omf file(s). Then it calls scrollkeeper-install, pointing
> > it to the doc(s) and omf file(s) it installed and giving it the path
> > $prefix/var/scrollkeeper.  If it is an RPM install, this path is something
> > like /tmp/scrollkeeper-0.1/var/scrollkeeper.
> 
> Just a technical detail: not really. There must be provisions in the
> software to make transparent the fact that during build it was in /tmp.
> 
> >  If it is a real install it
> > is /var/scrollkeeper.  We have a couple choices on how
> > scrollkeeper-install works.  I think the easiest is that if the database
> > doesn't exist, scrollkeeper just exits.
> 
> It will never exist before: remember, you are in tmp/.
> 
> >  In this case for a person
> > building an RPM, scrollkeeper-install looks for its main list,
> > /tmp/scrollkeeper-0.1/var/scrollkeeper/scrollkeeper-docs.  This file
> > doesn't exist, so scrollkeeper-install just exits.  If 'make install:' is
> > run by hand, scrollkeeper-install wants to append an entry to
> > /var/scrollkeeper/scrollkeeper-docs which exists, so scrollkeeper-install
> > proceeds happily updating the various database files in /var/scrollkeeper.
> > We will need $POST to have scrollkeeper-install.
> >
> > Another option would be to generate fresh database files under
> > /tmp/scrollkeeper-0.1/var/scrollkeeper if they don't exist already, but I
> > really don't see any point in doing this since they won't be installed and
> > they will be deleted a couple seconds later.
> 
> No, it would be the "usual" way of doing it: the freshly created
> database holds the partial (= only for this package) information and
> it's this partial database that is merged at postinstallation's time on
> the end-user's hard disk. Or even better, this partial database is only
> copied but then we have to cope with a database split into as many files
> as there are packages with OMF information.
> 
> > > Even if you could, it would be a bad idea. A "make install" instruction
> > > is supposed to be run equally from the end user's command line and from
> > > a packager's spec file.
> >
> > I'm not sure exactly what this means.  Does my paragraph above satisfy
> > this requirement?  If not, what would need to be changed?  Certainly it
> > would not be required that the real databases are changed.
> 
> Well, as far as scrollkeeper exits without producing an error code
> different than 0 when trying to "make install" if the database does not
> exist previously, then yes, you can assume that it can do whatever it
> wants, either not create a fresh one or create it. But it must also
> behave the same way on the user's hard disk if the database did not
> exist previously.
> 
> "make install" must behave the same way on the user's hard disk and on
> the packager's hard disk. If the database does not exist previously and
> if it exits then with an error code, then we have a problem.
> 
> --
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel

Re: [Scrollkeeper-devel] What about the distributions?

[Dan M. <d-...@uc...>]

Hi everybody,

First - I'm sorry I dropped off this list for a few days.  I have been
very busy.  The time I have been putting into ScrollKeeper was confined to
IRC, discussing these problems and possible solutions with Laszlo,
Jonathan, one of Red Hat's main packagers, and a couple other people.

I think we have resolved all the issues to having a main database (as
opposed to the mini-database idea, which I discuss below).  As Eric
suggests, we should merge the OMF files together in a pre-install step.  
This keeps things simple for the packager, since it is just 1 extra files
instead of N.  As part of 'make install', it adds the new information to
the big database.  For RPM *building*, it will be under BUILDROOT, which
means it creates a fresh database under BUILDROOT. In this "one big
database" approach, we do not keep the fresh database created under
BUILDROOT and install it.  We just have it get built to make sure no
errors occur, then it will be deleted.  Plus we can go snoop around in
there if we want to do testing.  The final package just gets the one new
big-OMF file.  It also gets a new line for %post and a new line for
%uninstall.

The other option is to have many mini-database files.  Each package
creates and installs a mini-database into a standard location, say
/var/lib/scrollkeeper/mini-dbs/ for the sake of discussion.  There are
many variations on how we can proceed from there, but I think the main
idea is to get rid of %post and %uninstall altogether.  So lets assume the
installation just installs the mini-db file.  As we discussed previously
on this list, merging the database files will take a certain amount of
time.  I'm guessing something like a second per package very roughly.  
(It has to read and parse the OMF file and all the docs in the package.)  
So to do this for 1000 packages would take a long time for a user to wait
to open his browser.  So, we would have to create cache files which are
incrementally added too (ie. the one big database above).  Now we have a
number of options on how we do things, but generally I see the following
problems:

1) When and how does the cache file (ie. big database) get created from
the smaller ones?  I think the previous suggestion was that whenever
scrollkeeper is queried for information, it first makes sure all the small
databases have been merged into the big cache database.  One big problem I
see with this is permissions and security.  If a user running
foo-help-browser asks scrollkeeper for the contents list and scrollkeeper
realizes it needs to update its big cache database from the small ones, it
will need to have permission to write to the scrollkeeper files.  This
means we run suid root programs, or create a special scrollkeeper user
which runs it.  The former seems like a security problem and could cause
problems with getting people to use and distribute scrollkeeper.  The
latter would work, but is a big pain since it means creating new entries
in /etc/password, /etc/shadow, /etc/groups, etc.  I don't even want to
think what will happen on NIS networks.  This seems to have its own set of
security issues and nasty complexities.  I think this would force us to
have each user have their own scrollkeeper database under
$HOME/.scrollkeeper/.  I think we will still have a huge time delay the
first time a user starts up their help browser.

2) Note that it also means that the first time to start up your
foo-help-browser after installing a few packages, you get a quite
noticable performance penalty.

3) It also does not provide feedback to the administrator.  Suppose the
sys admin installs foo which has a corrupt scrollkeeper database, or just
one which does not match the version of scrollkeeper on the machine.  It
installs fine, since it just copies the file.  The admin leaves, a user
comes along and runs the help browser, scrollkeeper tries to merge the
database... if things break, the sys admin is gone.  I think the whole
installation process should be done at install time, not half at install
time by the admin and half at run-time by the user.

4) Versioning becomes a very big problem.  Suppose a person is running
scrollkeeper 0.2.  If he installs a mini-db from a package where the build
machine was running scrollkeeper-0.1 or scrollkeeper-0.3, the mini-db will
potentially have the wrong format.  The best case scenario is scrollkeeper
ignores the file as if it didn't exist.  The worst case scenario is that
things break or get corrupted.

I am omitting one possibility - scrollkeeper upgrades old mini-dbs to the
new format.  I don't think we can do this because (1) the files it would
be modifying "belong" to the package foo, not to scrollkeeper, and (2) it
would not really be "updating" them. Instead it would have to go to the
original OMF file since the information it needs is not necessarily in the
old mini-db.  In this case, it is completely obsoleting the mini-db.  The
OMF file and the doc have all the information needed.  The mini-db just
has a subset of the information needed.  Plus it would not know how to
"downgrade" a mini-db since scrollkeeper-0.2 doesn't know what changed
between scrollkeeper-0.2 and scrollkeeper-0.3.

Generally, this means that a person cannot upgrade any packages on his
system (scrollkeeper or foo) away from original distributions packages
unless he can be sure the packager was using the same version of
scrollkeeper as the user has.  I think most Linux users would be very
upset by this.  (One of the best things about Linux is I can go out and
get the latest version of scrollkeeper or foo every month or two. Losing
this ability would be a big problem.)  Generally, it leads to even more
specificity of RPMs.  Right now, I can typically download an RPM for RH
6.0 or 6.1 or Mandrake and sometimes some others and they will run fine on
RH6.2.  Having the databases generated on the build machine will almost
guarantee to break this.  So the user is given a smaller set of packages
he can install and he is pressured into sticking with his distribution for
getting updates.

5) In most cases, it will be easier to add data into the single large
cache database by going directly to the OMF file and the doc than by
trying to use a mini-db.  In most cases, it is just a matter of writing
one set of parsing/writing functions instead of two.  (It is not a simple
concatenation of two files.)  In some cases, I'm not sure if it will even
be possible to "merge" databases unless we have a separate intermediate
format to the database.  This is basically because I can give instructions
to Bill on how to cut a piece of XML out of a file but Bob will not
necessarily know where to put the XML back in later unless I give Bob
additional information.

Sorry for the big list of problems, but I feel these are valid concerns.
Given my current understanding of the mini-database idea, I feel the
single database is a better solution.  The single database solution has
the extra complexity of having a post-install script, but I don't think
adding a couple trivial lines to a spec file is that bad.  I think it is
much simpler than the problems we have to address above and provides for
a more robust, secure, and flexible end-product.

Dan

BTW: There is a somewhat intermediate solution which involves having each
package foo add itself to a queue which should be processed by
scrollkeeper. It moves the database work onto the end-user machine, like
the 1-database solution. This solves the versioning problems discussed
above.  But it (optionally) gets rid of the post-install script.  This
could make the RPMs simpler (minus 4 lines), but leaves the security and
performance problems mentioned above. It also eliminates the extra coding
work needed for the extra conversion step in OMF/dock->min-db->db.  This
is very close to my original proposal to the Red Hat guys which they did
not seem to care for.

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Mercredi 25 Octobre 2000 17:57, vous avez écrit :
> 1) When and how does the cache file (ie. big database) get created from
> the smaller ones?  I think the previous suggestion was that whenever
> scrollkeeper is queried for information, it first makes sure all the small
> databases have been merged into the big cache database.

According to me, it's the best solution.

It unfortunately means that the first time doc is searched, all the small 
files built during installation distribution are merged and it can last a 
while. But it's no worst than the post-installation phase. 

>  One big problem I
> see with this is permissions and security.  If a user running
> foo-help-browser asks scrollkeeper for the contents list and scrollkeeper
> realizes it needs to update its big cache database from the small ones, it
> will need to have permission to write to the scrollkeeper files.  This
> means we run suid root programs,

Urgh

> or create a special scrollkeeper user
> which runs it.  The former seems like a security problem and could cause
> problems with getting people to use and distribute scrollkeeper.

Very true.

I might say something stupid, but why can't the contents list be predefined? 
Or is it imperative to build it out of OMF data?

Another solution, if the contents list is dynamic but small, is to create it 
in the user's home directory. It's a bit of redundant work to scrollkeeper to 
do this for each user, but it also enables each user to build a contents list
that matches the best its inline searches.

>  The
> latter would work, but is a big pain since it means creating new entries
> in /etc/password, /etc/shadow, /etc/groups, etc.  I don't even want to
> think what will happen on NIS networks.

;-)

>  This seems to have its own set of
> security issues and nasty complexities.  I think this would force us to
> have each user have their own scrollkeeper database under
> $HOME/.scrollkeeper/.

Hey we have our brains wired the same way ;-)

> 2) Note that it also means that the first time to start up your
> foo-help-browser after installing a few packages, you get a quite
> noticable performance penalty.

Wow the same thoughts ;-)

> 3) It also does not provide feedback to the administrator.  Suppose the
> sys admin installs foo which has a corrupt scrollkeeper database, or just
> one which does not match the version of scrollkeeper on the machine.  It
> installs fine, since it just copies the file.  The admin leaves, a user
> comes along and runs the help browser, scrollkeeper tries to merge the
> database... if things break, the sys admin is gone.  I think the whole
> installation process should be done at install time, not half at install
> time by the admin and half at run-time by the user.

But at install time it means postinstall scripts. It's technically better, 
but I can't figure out how distributions will react. It was painful enough to 
change to LSB directory tree at Caldera.

> 4) Versioning becomes a very big problem.  Suppose a person is running
> scrollkeeper 0.2.  If he installs a mini-db from a package where the build
> machine was running scrollkeeper-0.1 or scrollkeeper-0.3, the mini-db will
> potentially have the wrong format.  The best case scenario is scrollkeeper
> ignores the file as if it didn't exist.  The worst case scenario is that
> things break or get corrupted.

This problem you will get anyway with OMF files.

The solution is the database format used in the database. If it's 
XML-conformant then it's much easier to have different XML formats
in the same database. XML allows backward and forward compatibility if your 
changes respect very minimalistic rules. For example you can add and remove 
tags.

> I am omitting one possibility - scrollkeeper upgrades old mini-dbs to the
> new format.  I don't think we can do this because (1) the files it would
> be modifying "belong" to the package foo, not to scrollkeeper, and (2) it
> would not really be "updating" them. Instead it would have to go to the
> original OMF file since the information it needs is not necessarily in the
> old mini-db.  In this case, it is completely obsoleting the mini-db.  The
> OMF file and the doc have all the information needed.  The mini-db just
> has a subset of the information needed.  Plus it would not know how to
> "downgrade" a mini-db since scrollkeeper-0.2 doesn't know what changed
> between scrollkeeper-0.2 and scrollkeeper-0.3.
>
> Generally, this means that a person cannot upgrade any packages on his
> system (scrollkeeper or foo) away from original distributions packages
> unless he can be sure the packager was using the same version of
> scrollkeeper as the user has.  I think most Linux users would be very
> upset by this.  (One of the best things about Linux is I can go out and
> get the latest version of scrollkeeper or foo every month or two. Losing
> this ability would be a big problem.)  Generally, it leads to even more
> specificity of RPMs.  Right now, I can typically download an RPM for RH
> 6.0 or 6.1 or Mandrake and sometimes some others and they will run fine on
> RH6.2.  Having the databases generated on the build machine will almost
> guarantee to break this.  So the user is given a smaller set of packages
> he can install and he is pressured into sticking with his distribution for
> getting updates.
>
> 5) In most cases, it will be easier to add data into the single large
> cache database by going directly to the OMF file and the doc than by
> trying to use a mini-db.

Again, post-install scripts *are* the technical solution, but will upset the 
distributions. There's no need for you to prove that they are better, we 
already know it.

>  In most cases, it is just a matter of writing
> one set of parsing/writing functions instead of two.  (It is not a simple
> concatenation of two files.)  In some cases, I'm not sure if it will even
> be possible to "merge" databases unless we have a separate intermediate
> format to the database.  This is basically because I can give instructions
> to Bill on how to cut a piece of XML out of a file but Bob will not
> necessarily know where to put the XML back in later unless I give Bob
> additional information.
>
> Sorry for the big list of problems, but I feel these are valid concerns.

Yes. I think we are in the heart of the problem. Sorry to have started all of 
this, but it was a mere fact than in a distribution a "make install" is done 
by the packager, not by the end user.

> Given my current understanding of the mini-database idea, I feel the
> single database is a better solution.

It is. No discussion. If we had a way of not having the packagers explicitely 
write the calls to those postinstall and postuninstall scripts it would be 
perfect.

>  The single database solution has
> the extra complexity of having a post-install script, but I don't think
> adding a couple trivial lines to a spec file is that bad.  I think it is

A post-install script is not something complicated. It is generating a 
problem with the human factor. Those lines have to be added in about a 
thousand spec files in each distribution and the documentation is not usually 
the top priority anywhere.

Let me state the problem the other way round. We won't merge the OMF nor the 
DocBook files either. So why should we merge the extra data and not these 
ones?

> much simpler than the problems we have to address above and provides for
> a more robust, secure, and flexible end-product.

Sure.

> BTW: There is a somewhat intermediate solution which involves having each
> package foo add itself to a queue which should be processed by
> scrollkeeper.

And how does it add itself to the queue? In the post-install script? ;-)

This adds nothing new to the choice between post-install and mini-databases.

> It moves the database work onto the end-user machine, like
> the 1-database solution. This solves the versioning problems discussed
> above.  But it (optionally) gets rid of the post-install script.  This
> could make the RPMs simpler (minus 4 lines), but leaves the security and
> performance problems mentioned above. It also eliminates the extra coding
> work needed for the extra conversion step in OMF/dock->min-db->db.  This
> is very close to my original proposal to the Red Hat guys which they did
> not seem to care for.

I'm not sure I understand what you propose here.

-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Dan M. <d-...@uc...>]

On Fri, 27 Oct 2000, Eric Bischoff wrote:

> Le Mercredi 25 Octobre 2000 17:57, vous avez écrit :
> > 1) When and how does the cache file (ie. big database) get created from
> > the smaller ones?  I think the previous suggestion was that whenever
> > scrollkeeper is queried for information, it first makes sure all the small
> > databases have been merged into the big cache database.
> 
> According to me, it's the best solution.
> 
> It unfortunately means that the first time doc is searched, all the small 
> files built during installation distribution are merged and it can last a 
> while. But it's no worst than the post-installation phase. 

It is worse, because an admin is willing to wait five seconds or more for
a package to install, but users don't want to wait 5 extra seconds when
they open their help browser.

> >  One big problem I
> > see with this is permissions and security.  If a user running
> > foo-help-browser asks scrollkeeper for the contents list and scrollkeeper
> > realizes it needs to update its big cache database from the small ones, it
> > will need to have permission to write to the scrollkeeper files.  This
> > means we run suid root programs,
> 
> Urgh
> 
> > or create a special scrollkeeper user
> > which runs it.  The former seems like a security problem and could cause
> > problems with getting people to use and distribute scrollkeeper.
> 
> Very true.
> 
> I might say something stupid, but why can't the contents list be predefined? 
> Or is it imperative to build it out of OMF data?

I'm not sure I follow your question.  In order to build the contents list,
you have to know which packages are installed on the computer.  There is
no way of knowing this until you are on the user's machine.  Further, it
would be very nice if the categorization list of the contents list was on
the user's machine.  This way an admin could choose not to have certain
categories in the contents list.  For example if I was an admin for a set
of machines which I knew newbies were using, I might chop things like man
pages or LDP HOWTOs out of the Contents List since they would only confuse
newbies further.  I might also want to add my own categories for fields of
interest for my users.

> Another solution, if the contents list is dynamic but small, is to create it 
> in the user's home directory. It's a bit of redundant work to scrollkeeper to 
> do this for each user, but it also enables each user to build a contents list
> that matches the best its inline searches.

I am hesitant to do this, partly because it seems strange and partly
because it will cause a performance problem. At the same time, I like the
idea of having this as an option for the future since it would potentially
allow each user to configure their own Contents List, which would be
great.  I think the default behavior should not be to replicate itself
into each user's $HOME directory, but to use the system-wide database
unless a user explicitly wants to add or remove contents.  This would be
feature which we would not implement for a long time anyway.  If and when
we implement it, we can probably get around the performance problem by
copying the system-wide database to start with.  However not having a
system-wide database means that each user gets completely hammered the
first time they open their browser.

> >  The
> > latter would work, but is a big pain since it means creating new entries
> > in /etc/password, /etc/shadow, /etc/groups, etc.  I don't even want to
> > think what will happen on NIS networks.
> 
> ;-)
> 
> >  This seems to have its own set of
> > security issues and nasty complexities.  I think this would force us to
> > have each user have their own scrollkeeper database under
> > $HOME/.scrollkeeper/.
> 
> Hey we have our brains wired the same way ;-)

:)

> > 2) Note that it also means that the first time to start up your
> > foo-help-browser after installing a few packages, you get a quite
> > noticable performance penalty.
> 
> Wow the same thoughts ;-)
> 
> > 3) It also does not provide feedback to the administrator.  Suppose the
> > sys admin installs foo which has a corrupt scrollkeeper database, or just
> > one which does not match the version of scrollkeeper on the machine.  It
> > installs fine, since it just copies the file.  The admin leaves, a user
> > comes along and runs the help browser, scrollkeeper tries to merge the
> > database... if things break, the sys admin is gone.  I think the whole
> > installation process should be done at install time, not half at install
> > time by the admin and half at run-time by the user.
> 
> But at install time it means postinstall scripts. It's technically better, 
> but I can't figure out how distributions will react. It was painful enough to 
> change to LSB directory tree at Caldera.

I agree this is a very important issue.  We have to be confident the
distributions will accept this.  However changing the directory tree is a
*much* bigger concern than adding scrollkeeper.  Plus, scrollkeeper can be
added in little pieces.  Scrollkeeper will be adopted slowly, giving
distributions time to get familiar with it.  If GNOME 1.4 can use it, as I
am desperately hoping, then that means that maybe 20 packages will be
using it in the next few months.  (GNOME 1.4 should be out in about 6
weeks, if it can stay on schedule ;)  If KDE adopts it for their next
release along with a few other people, we are looking at maybe 70 packages
5 or 6 months from now.  That isn't so much to really swamp anybody.  
Plus, I'm planning that the lines we add to the spec files are completely
generic.  Basically just cut and paste 4 lines into the spec file and you
are done.  If you forget, then that doc doesn't show up in the contents
list. It doesn't really break anything, so I don't think anybody is going
to scream about a distribution forgetting to add it to a couple spec
files.

I think it would help a lot to actually try to implement this
solution.  Perhaps it will turn out to be more complicated than I am
thinking?

> > 4) Versioning becomes a very big problem.  Suppose a person is running
> > scrollkeeper 0.2.  If he installs a mini-db from a package where the build
> > machine was running scrollkeeper-0.1 or scrollkeeper-0.3, the mini-db will
> > potentially have the wrong format.  The best case scenario is scrollkeeper
> > ignores the file as if it didn't exist.  The worst case scenario is that
> > things break or get corrupted.
> 
> This problem you will get anyway with OMF files.
> 
> The solution is the database format used in the database. If it's 
> XML-conformant then it's much easier to have different XML formats
> in the same database. XML allows backward and forward compatibility if your 
> changes respect very minimalistic rules. For example you can add and remove 
> tags.

XML is great, I agree.  However in this case there is a much better
solution than just having XML give us the ability to not have information.  
If the user upgrades from scrollkeeper-0.1 to scrollkeeper-0.2 (or has
scrollkeeper-0.2 and downloads a package built with scrollkeeper-0.1),
they could keep their old XML mini-databases, scrollkeeper can be made not
to crash, and the new features in scrollkeeper-0.2 just won't work on the
older mini-databases. So the new features will not really be very usable
for 6 months or so until most packages have new releases.  However it is a
huge shame to use a solution like this when we know that all the
information we need is sitting in the original SGML and OMF files on the
hard drive and scrollkeeper knows exactly where they are at.  If I were
the scrollkeeper, I would say "Forget this obsolete mini-db, I know where
the information is at and I will go extract it and merge it into my real
database":)

> > I am omitting one possibility - scrollkeeper upgrades old mini-dbs to the
> > new format.  I don't think we can do this because (1) the files it would
> > be modifying "belong" to the package foo, not to scrollkeeper, and (2) it
> > would not really be "updating" them. Instead it would have to go to the
> > original OMF file since the information it needs is not necessarily in the
> > old mini-db.  In this case, it is completely obsoleting the mini-db.  The
> > OMF file and the doc have all the information needed.  The mini-db just
> > has a subset of the information needed.  Plus it would not know how to
> > "downgrade" a mini-db since scrollkeeper-0.2 doesn't know what changed
> > between scrollkeeper-0.2 and scrollkeeper-0.3.
> >
> > Generally, this means that a person cannot upgrade any packages on his
> > system (scrollkeeper or foo) away from original distributions packages
> > unless he can be sure the packager was using the same version of
> > scrollkeeper as the user has.  I think most Linux users would be very
> > upset by this.  (One of the best things about Linux is I can go out and
> > get the latest version of scrollkeeper or foo every month or two. Losing
> > this ability would be a big problem.)  Generally, it leads to even more
> > specificity of RPMs.  Right now, I can typically download an RPM for RH
> > 6.0 or 6.1 or Mandrake and sometimes some others and they will run fine on
> > RH6.2.  Having the databases generated on the build machine will almost
> > guarantee to break this.  So the user is given a smaller set of packages
> > he can install and he is pressured into sticking with his distribution for
> > getting updates.
> >
> > 5) In most cases, it will be easier to add data into the single large
> > cache database by going directly to the OMF file and the doc than by
> > trying to use a mini-db.
> 
> Again, post-install scripts *are* the technical solution, but will upset the 
> distributions. There's no need for you to prove that they are better, we 
> already know it.

My talks with Jonathan and Jeff (via. Jonathan) at Red Hat have indicated
that they did not feel this would place an undo burden on distributors.  
That is my impression as well, although you (Eric), Jonathan, and Jeff
have much more expertise in this area than I do.

> >  In most cases, it is just a matter of writing
> > one set of parsing/writing functions instead of two.  (It is not a simple
> > concatenation of two files.)  In some cases, I'm not sure if it will even
> > be possible to "merge" databases unless we have a separate intermediate
> > format to the database.  This is basically because I can give instructions
> > to Bill on how to cut a piece of XML out of a file but Bob will not
> > necessarily know where to put the XML back in later unless I give Bob
> > additional information.
> >
> > Sorry for the big list of problems, but I feel these are valid concerns.
> 
> Yes. I think we are in the heart of the problem. Sorry to have started all of 
> this, but it was a mere fact than in a distribution a "make install" is done 
> by the packager, not by the end user.

It is great to have your help.  My original plan would have had many OMF
files in each spec file and many calls to post-install scripts, which
would have made quite a bit of work for packagers.

The difference in configuration between the packager's machine and the
end-user's machine is a big part of why I prefer the post-install script
solution.

> > Given my current understanding of the mini-database idea, I feel the
> > single database is a better solution.
> 
> It is. No discussion. If we had a way of not having the packagers explicitely 
> write the calls to those postinstall and postuninstall scripts it would be 
> perfect.

I am hoping that we can make this process very simple for packagers (as
well as for developers).  I am confident that we can make packagers' lives
prety simple here while still using post-install scripts.  We haven't
discussed the developers' job on this list yet, although I have done so
somewhat with Laszlo in private.  I'll bring up the one issue which sticks
out as tricky right now:

Suppose we have a package with two apps, foo1 and foo2, each with a
doc.  So we have:
foo1/docs/foo1-manual.xml
foo1/docs/foo1-manual.omf
foo2/docs/foo2-manual.xml
foo2/docs/foo2-manual.omf

Now we want to use the pre-install script as you suggested.  This script
should basically do two things:
1) Allow us to set the URL in the OMF file to point to where the doc will
be installed. This is relatively straightforward, so I won't talk about it
any more right now.
2) Merge the OMF files into one big OMF file.

While #2 seems relatively trivial (we could almost do a "cat <filename> >>
<bigfilename>" where <bigfilename> will probably need to be the relative
path to the file with some "../.." in it), I'm not sure how to get it to
work right with the build process and autoconf.  For simplicity, imagine
we use 'cat' as above.  We would like to make sure that if somebody does
'make' twice in a row, it does not just keep appending the same entries
into <bigfilename>, duplicating them.  It would seem that we want to
delete <bigfilename> each time we start a 'make', but I don't see how we
can do this with autoconf/automake.  (Anybody know?)  Another solutions
would be to have scrollkeeper-preinstall be smart enough to know if it is
duplicating an entry and just update instead of add an all new entry.  
This would not delete an entry if one was removed.  Another option would
be to assume that people always do 'make clean' between makes, but I
imagine that builders and developers monkey around a fair amount without
doing a 'make clean' so this may be a bad assumption.  Any ideas for a
solution to this one?  Remember - it has to be absolutely trivial for
developers and packagers.

> >  The single database solution has
> > the extra complexity of having a post-install script, but I don't think
> > adding a couple trivial lines to a spec file is that bad.  I think it is
> 
> A post-install script is not something complicated. It is generating a 
> problem with the human factor. Those lines have to be added in about a 
> thousand spec files in each distribution and the documentation is not usually 
> the top priority anywhere.

Well, let's say about 50 or 100 packages maximum in the next year or
so.  I think the combination of making it very easy and the fact that an
infrequent ommission will not have very serious consequences will help
too.

> Let me state the problem the other way round. We won't merge the OMF nor the 
> DocBook files either. So why should we merge the extra data and not these 
> ones?

Aside from all the reasons I've already stated? ;)

The docs and OMF files belong to the individual packages to install and
uninstall as necessary to remain in sync with the packages.  The
scrollkeeper database belongs to scrollkeeper, to maintain, update,
synchronize, modify, and use as scrollkeeper needs.  I think having
scrollkeeper's database created, shipped, and maintained (or in this case
*not maintained*) by the applications (in many pieces) instead of
by scrollkeeper greatly restricts scrollkeeper's ability to do its job.

> > much simpler than the problems we have to address above and provides for
> > a more robust, secure, and flexible end-product.
> 
> Sure.
> 
> > BTW: There is a somewhat intermediate solution which involves having each
> > package foo add itself to a queue which should be processed by
> > scrollkeeper.
> 
> And how does it add itself to the queue? In the post-install script? ;-)
> 
> This adds nothing new to the choice between post-install and mini-databases.
> 
> > It moves the database work onto the end-user machine, like
> > the 1-database solution. This solves the versioning problems discussed
> > above.  But it (optionally) gets rid of the post-install script.  This
> > could make the RPMs simpler (minus 4 lines), but leaves the security and
> > performance problems mentioned above. It also eliminates the extra coding
> > work needed for the extra conversion step in OMF/dock->min-db->db.  This
> > is very close to my original proposal to the Red Hat guys which they did
> > not seem to care for.
> 
> I'm not sure I understand what you propose here.

Well, I prefer the post-install script, which is why I do not emphasize
this solution.  However, if we *really* wanted to avoid the script, we
could do something like: Have a magic directory every package installs a
file into, say /var/lib/scrollkeeper/packages.  Each package installs a
short file in here which lists the full path to the OMF file(s) installed
by the package.  Then whenever scrollkeeper is run it checks to see if
there are any new, modified, or missing files in this directory.  If so,
it updates its database.  The advantage is that it still uses one database
making things simpler and it can keep the versions correct.  The downside
is that it still has the permissions and performance problem, it requires
all packages to install something into a special directory, and it is
generally hokey.  Plus it has to detect when a file has been modified,
which means either comparing the contents or checking timestamps, neither
of which seem pretty.

Dan

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Dimanche 29 Octobre 2000 23:10, Dan Mueth a écrit :
> On Fri, 27 Oct 2000, Eric Bischoff wrote:
> It is worse, because an admin is willing to wait five seconds or more for
> a package to install, but users don't want to wait 5 extra seconds when
> they open their help browser.

Yes.

Well the 5 extra seconds will be on the *first* time they open the browser. 
The next times it will be much quicker. You don't install tons of packages 
every day.

> > Another solution, if the contents list is dynamic but small, is to create
> > it in the user's home directory. It's a bit of redundant work to
> > scrollkeeper to do this for each user, but it also enables each user to
> > build a contents list that matches the best its inline searches.
>
> I am hesitant to do this, partly because it seems strange and partly
> because it will cause a performance problem. At the same time, I like the
> idea of having this as an option for the future since it would potentially
> allow each user to configure their own Contents List, which would be
> great.  I think the default behavior should not be to replicate itself
> into each user's $HOME directory, but to use the system-wide database
> unless a user explicitly wants to add or remove contents.

Yes.

>  This would be
> feature which we would not implement for a long time anyway.  If and when
> we implement it, we can probably get around the performance problem by
> copying the system-wide database to start with.  However not having a
> system-wide database means that each user gets completely hammered the
> first time they open their browser.

Yup.

> > But at install time it means postinstall scripts. It's technically
> > better, but I can't figure out how distributions will react. It was
> > painful enough to change to LSB directory tree at Caldera.
>
> I agree this is a very important issue.  We have to be confident the
> distributions will accept this.  However changing the directory tree is a
> *much* bigger concern than adding scrollkeeper.

Are you sure? If you count the changed lines in the SPEC files it would even 
be worse.

>  Plus, scrollkeeper can be
> added in little pieces.  Scrollkeeper will be adopted slowly, giving
> distributions time to get familiar with it.  If GNOME 1.4 can use it, as I
> am desperately hoping, then that means that maybe 20 packages will be
> using it in the next few months.  (GNOME 1.4 should be out in about 6
> weeks, if it can stay on schedule ;)  If KDE adopts it for their next
> release along with a few other people, we are looking at maybe 70 packages
> 5 or 6 months from now.  That isn't so much to really swamp anybody.
> Plus, I'm planning that the lines we add to the spec files are completely
> generic.  Basically just cut and paste 4 lines into the spec file and you
> are done.  If you forget, then that doc doesn't show up in the contents
> list. It doesn't really break anything, so I don't think anybody is going
> to scream about a distribution forgetting to add it to a couple spec
> files.

Hmmm if your help browser "forgets" to present you with the results of your 
query it's really bad - once you have a tool you start trusting it.

> > Again, post-install scripts *are* the technical solution, but will upset
> > the distributions. There's no need for you to prove that they are better,
> > we already know it.
>
> My talks with Jonathan and Jeff (via. Jonathan) at Red Hat have indicated
> that they did not feel this would place an undo burden on distributors.
> That is my impression as well, although you (Eric), Jonathan, and Jeff
> have much more expertise in this area than I do.

> > Yes. I think we are in the heart of the problem. Sorry to have started
> > all of this, but it was a mere fact than in a distribution a "make
> > install" is done by the packager, not by the end user.
>
> It is great to have your help.  My original plan would have had many OMF
> files in each spec file and many calls to post-install scripts, which
> would have made quite a bit of work for packagers.

Even if we stick to the post-install script solution, which I do not 
encourage, it's true we did a lot of progress on this front.

> > Let me state the problem the other way round. We won't merge the OMF nor
> > the DocBook files either. So why should we merge the extra data and not
> > these ones?
>
> Aside from all the reasons I've already stated? ;)
>
> The docs and OMF files belong to the individual packages to install and
> uninstall as necessary to remain in sync with the packages.  The
> scrollkeeper database belongs to scrollkeeper, to maintain, update,
> synchronize, modify, and use as scrollkeeper needs.  I think having
> scrollkeeper's database created, shipped, and maintained (or in this case
> *not maintained*) by the applications (in many pieces) instead of
> by scrollkeeper greatly restricts scrollkeeper's ability to do its job.

Do we have a real evaluation of the cost of having a split database?

> Well, I prefer the post-install script, which is why I do not emphasize
> this solution.  However, if we *really* wanted to avoid the script, we
> could do something like: Have a magic directory every package installs a
> file into, say /var/lib/scrollkeeper/packages.  Each package installs a
> short file in here which lists the full path to the OMF file(s) installed
> by the package.  Then whenever scrollkeeper is run it checks to see if
> there are any new, modified, or missing files in this directory.  If so,
> it updates its database.

It's what I'm suggesting for ages. The only real drawback with this solution 
is the delay when you first need the merged database. And I'm not sure it 
will be 5 seconds. Of course there are the other problems like the rights but 
we know there are solutions.

>  The advantage is that it still uses one database
> making things simpler and it can keep the versions correct.  The downside
> is that it still has the permissions and performance problem, it requires
> all packages to install something into a special directory, and it is
> generally hokey.  Plus it has to detect when a file has been modified,
> which means either comparing the contents or checking timestamps, neither
> of which seem pretty.


-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Thu, 26 Oct 2000, Eric Bischoff wrote:

> Ali Abdin wrote:
> > 
> > 
> > Perhaps what we need are two directories or something. So you would have:
> > prefix/var/lib/scrollkeeper/processed
> > prefix/var/lib/scrollkeeper/unprocessed
> 
> This is great!!!! :-)
> 
> a) during "make install"
> "scrollkeeper --install" will stores the partial information in either
> /var/lib/scrollkeeper/unprocessed (end-user doing "make install")
> or in
> /tmp/var/lib/scrollkeeper/unprocessed (packager building the package)
> 
> b) during RPM installation
> then the unprocessed file is simply copied into
> /var/lib/scrollkeeper/unprocessed
> 
> c) when the help browser starts
> it is requested to launch "scrollkeeper --update". This one sees that
> they are unprocessed files,
> merges them into the real database, and removes them from the
> /unprocessed directory.

No this really really sucks actually. App authors shouldn't be forced to 
execute scrollkeeper --update everytime somebody wishes to browse a help 
file.

The decent options to me (for this approach) would be to:
1) Use a cron script
2) Make scrollkeeper a daemon
 
> In fact the "unprocessed" directory should be two directories:
> 
> 	/var/lib/scrollkeeper/toinstall
> 	/var/lib/scrollkeeper/touninstall
> 
> because we're having the same problem during uninstallation.
> 
> > An "initial" scrollkeeper-install would put the file in the unprocessed
> > section, scrollkeeper could then at some designated time, check all files in
> > the unprocessed section, if they are found on the local drive they are moved
> > to processed, otherwise they are removed from unprocessed.
> > 
> > But this presents another issue. What about "online docs"? the above
> > implementation would force people to have docs on the "local" path!!! Instead
> > of deleting the file, perhaps we should move it into a "not-installed"
> > directory. That way if the help file is located at
> > "http://www.doc.com/file.sgml" We could access it from the not-installed
> > section.
> > 
> > The problem would arise with "syncing" the two directories (what if you have a
> > "not-installed" doc and then later install it locally)
> > 
> > So perhaps we shouldn't have multiple databases (or directories, or whatever
> > you want to call it). Perhaps we should always install files...
> > 
> > If a doc is not found/installed locally then ScrollKeeper should handle it.
> > This sounds a lot like what Eric was saying doesn't it? ;) But he makes it
> > sound that applications (help browsers) should handle it. Instead, I think
> > that ScrollKeeper should natively handle it.
> > 
> > If I ask the ScrollKeeper database for "all installed docs" it would return
> > that - I /could/ ask for all "not-installed" docs (which would return these
> > docs + the HTTP ones + FTP ones + etc...)
> > 
> > Generally, when a help browser displays its "Index" (for Nautilus that is the
> > Help Sidebar) it would only display the "installed" docs...
> > 
> > What do you think of this proposal?
> 
> Great. I'm not sure about the online docs, but for the local docs it's
> really a good idea.
> 
> -- 
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/
> 

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Le Jeudi 26 Octobre 2000 10:27, vous avez écrit :
>
> No this really really sucks actually. App authors shouldn't be forced to
> execute scrollkeeper --update everytime somebody wishes to browse a help
> file.
>
> The decent options to me (for this approach) would be to:
> 1) Use a cron script
> 2) Make scrollkeeper a daemon

These are also options, sure.
-- 
Éric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Mon, 13 Nov 2000, Eric Bischoff wrote:

> Le Samedi 28 Octobre 2000 00:38, Ali Abdin a écrit :
> > > This will probably be autodetected, thanks to Unix rights system. A
> > > package is always built as a normal user the first time. The usual
> > > process is then to add the right arguments for buildrooting in the spec
> > > file.
> >
> > No it won't - most SPEC files do not do this!!
> 
> I suppose this is a misunderstanding. Yes they do. They need it at least to 
> pass the build root to the make files.

No - most specfiles do not specify a '--localstatedir' - so the autoconf 
uses the "default" by assuming it is the same as --prefix (i.e. usually 
/usr or /usr/local)

Regards,
Ali

Re: [Scrollkeeper-devel] What about the distributions?

[Ali A. <ALI...@au...>]

On Mon, 13 Nov 2000, Eric Bischoff wrote:

> Le Dimanche 29 Octobre 2000 23:10, Dan Mueth a écrit :
> > On Fri, 27 Oct 2000, Eric Bischoff wrote:
> > It is worse, because an admin is willing to wait five seconds or more for
> > a package to install, but users don't want to wait 5 extra seconds when
> > they open their help browser.
> 
> Yes.
> 
> Well the 5 extra seconds will be on the *first* time they open the browser. 
> The next times it will be much quicker. You don't install tons of packages 
> every day.

Those 5 seconds are unacceptable to me. And they won't be done only the 
*first* time, they will be done everytime you install a new doc.
 
> > > Another solution, if the contents list is dynamic but small, is to create
> > > it in the user's home directory. It's a bit of redundant work to
> > > scrollkeeper to do this for each user, but it also enables each user to
> > > build a contents list that matches the best its inline searches.
> >
> > I am hesitant to do this, partly because it seems strange and partly
> > because it will cause a performance problem. At the same time, I like the
> > idea of having this as an option for the future since it would potentially
> > allow each user to configure their own Contents List, which would be
> > great.  I think the default behavior should not be to replicate itself
> > into each user's $HOME directory, but to use the system-wide database
> > unless a user explicitly wants to add or remove contents.
> 
> Yes.
> 
> >  This would be
> > feature which we would not implement for a long time anyway.  If and when
> > we implement it, we can probably get around the performance problem by
> > copying the system-wide database to start with.  However not having a
> > system-wide database means that each user gets completely hammered the
> > first time they open their browser.
> 
> Yup.
> 
> > > But at install time it means postinstall scripts. It's technically
> > > better, but I can't figure out how distributions will react. It was
> > > painful enough to change to LSB directory tree at Caldera.
> >
> > I agree this is a very important issue.  We have to be confident the
> > distributions will accept this.  However changing the directory tree is a
> > *much* bigger concern than adding scrollkeeper.
> 
> Are you sure? If you count the changed lines in the SPEC files it would even 
> be worse.

I do not think that changing one line in several thousand spec files is a 
big issue. Its not like they are done by "one person". 

I agree that it would be nice to avoid thise post-install script. But the 
alternatives that have been suggested are not feasibly (in my opinion)
 
> >  Plus, scrollkeeper can be
> > added in little pieces.  Scrollkeeper will be adopted slowly, giving
> > distributions time to get familiar with it.  If GNOME 1.4 can use it, as I
> > am desperately hoping, then that means that maybe 20 packages will be
> > using it in the next few months.  (GNOME 1.4 should be out in about 6
> > weeks, if it can stay on schedule ;)  If KDE adopts it for their next
> > release along with a few other people, we are looking at maybe 70 packages
> > 5 or 6 months from now.  That isn't so much to really swamp anybody.
> > Plus, I'm planning that the lines we add to the spec files are completely
> > generic.  Basically just cut and paste 4 lines into the spec file and you
> > are done.  If you forget, then that doc doesn't show up in the contents
> > list. It doesn't really break anything, so I don't think anybody is going
> > to scream about a distribution forgetting to add it to a couple spec
> > files.
> 
> Hmmm if your help browser "forgets" to present you with the results of your 
> query it's really bad - once you have a tool you start trusting it.

It doesn't "forget" - it just doesn't mention the fact that your help 
file is installed because the OMF file is not installed. Somebody will 
then complain "this app doesn't have a help file". The author would then 
properly install the OMF file.

> > > Again, post-install scripts *are* the technical solution, but will upset
> > > the distributions. There's no need for you to prove that they are better,
> > > we already know it.
> >
> > My talks with Jonathan and Jeff (via. Jonathan) at Red Hat have indicated
> > that they did not feel this would place an undo burden on distributors.
> > That is my impression as well, although you (Eric), Jonathan, and Jeff
> > have much more expertise in this area than I do.
> 
> > > Yes. I think we are in the heart of the problem. Sorry to have started
> > > all of this, but it was a mere fact than in a distribution a "make
> > > install" is done by the packager, not by the end user.
> >
> > It is great to have your help.  My original plan would have had many OMF
> > files in each spec file and many calls to post-install scripts, which
> > would have made quite a bit of work for packagers.
> 
> Even if we stick to the post-install script solution, which I do not 
> encourage, it's true we did a lot of progress on this front.

I'd like to know "other" people's opinion on the post-install script 
thing (so far its mainly been a "GNOME-affiliated camp" vs. 
"KDE-affiliated camp" debate). What about the various other people's 
opinions? ;)
 
> > > Let me state the problem the other way round. We won't merge the OMF nor
> > > the DocBook files either. So why should we merge the extra data and not
> > > these ones?
> >
> > Aside from all the reasons I've already stated? ;)
> >
> > The docs and OMF files belong to the individual packages to install and
> > uninstall as necessary to remain in sync with the packages.  The
> > scrollkeeper database belongs to scrollkeeper, to maintain, update,
> > synchronize, modify, and use as scrollkeeper needs.  I think having
> > scrollkeeper's database created, shipped, and maintained (or in this case
> > *not maintained*) by the applications (in many pieces) instead of
> > by scrollkeeper greatly restricts scrollkeeper's ability to do its job.
> 
> Do we have a real evaluation of the cost of having a split database?
> 
> > Well, I prefer the post-install script, which is why I do not emphasize
> > this solution.  However, if we *really* wanted to avoid the script, we
> > could do something like: Have a magic directory every package installs a
> > file into, say /var/lib/scrollkeeper/packages.  Each package installs a
> > short file in here which lists the full path to the OMF file(s) installed
> > by the package.  Then whenever scrollkeeper is run it checks to see if
> > there are any new, modified, or missing files in this directory.  If so,
> > it updates its database.
> 
> It's what I'm suggesting for ages. The only real drawback with this solution 
> is the delay when you first need the merged database. And I'm not sure it 
> will be 5 seconds. Of course there are the other problems like the rights but 
> we know there are solutions.
> 
> >  The advantage is that it still uses one database
> > making things simpler and it can keep the versions correct.  The downside
> > is that it still has the permissions and performance problem, it requires
> > all packages to install something into a special directory, and it is
> > generally hokey.  Plus it has to detect when a file has been modified,
> > which means either comparing the contents or checking timestamps, neither
> > of which seem pretty.
> 
> 
> -- 
> Éric Bischoff  -  Documentation and Localization
> Caldera (Deutschland) GmbH - Linux for eBusiness
> Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
> http://www.caldera.de/
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel
> 

Re: [Scrollkeeper-devel] What about the distributions?

[laszlo k. <las...@su...>]

> Unfortunately - when people './configure;make;make install' they will
> miss out on this aspect.
> 
> What you might need to do is to create some sort of
> system/cron-job/daemon in which scrollkeeper checks the local files
> installed - if it is missing it will remove them from its internal
> database.
> 
> Otherwise you will need a big huge README in each project saying "please
> run scrollkeeper-install after 'make install' is done"

Can each project have in its Makefile a section that checks for
Scrollkeeper and if it's installed then installs the docs for
Scrollkeeper also (same for uninstall)? I guess the problem is the large
number of packages and distributors, but it can be done in time if the
system (Scrollkeeper and the Help Browser) is good so distributors see
it is better to support it.

Laszlo

Re: [Scrollkeeper-devel] What about the distributions?

[Eric B. <eb...@cy...>]

Ali Abdin wrote:
> 
> On Wed, 18 Oct 2000, Eric Bischoff wrote:
> 
> 
> Unfortunately - when people './configure;make;make install' they will
> miss out on this aspect.

You're right. This means the line in "install:" section should be enough
for updating the database on user's hard disk.

But you must also provide a mechanism to reflect those changes in the
database on packager's hard disk to the end-user's hard disk at
installation time.

Otherwise here is what will happen:

The packager installs the software on his hard disk. His database will
be created (since usually people package in /tmp, the database will be
created from scratch and only hold the entries for the software that is
being packaged).

So at the end the packager sees a new file, let's say
/tmp/usr/share/scrollkeeper/omf.xml. He has three choices:
- ignore the file, it means that he doesn't care about scrollkeeper or
doesn't want to or doesn't understand it or is too lazy to ask himself
the question... :-(
- put it into the %Files section of the specfile (or in the tgz on a
slackware). As soon as two packages use scrollkeeper there will be a
conflict UNLESS YOU GIVE EACH XML FILE A DIFFERENT, PACKAGE-RELATED NAME
- use a post-installation script that will merge this file with the
user's database on his hard disk.

So there are two opportunities for us:
1) Choose a naming convention that will ensure that each package has its
XML data stored in a separate file with a name of its own
2) Provide a kind of "scrollkeeper-merge" utility that will merge the
file on the packager's hard disk 

The first solution may hinder performance at database's browsing time.
The second solution means more work for the distributions :-( and may
hinder the distributions installlation time (post-installation scripts
are usually slow).

If those issues were discussed already, I apologize in advance, I had to
read quickly many old emails to get in pace with this list. Otherwise, I
think this issue is wort considering it.

> What you might need to do is to create some sort of
> system/cron-job/daemon in which scrollkeeper checks the local files
> installed - if it is missing it will remove them from its internal
> database.
>
> Otherwise you will need a big huge README in each project saying "please
> run scrollkeeper-install after 'make install' is done"

No, you're right, none of these are acceptable. We must update the
database in the Makefile AND provide a mechanism to packagers, should it
be have a clear naming convention for the slices of the database or a
post-installation script.

-- 
 Éric Bischoff   -   mailto:eb...@cy...
 __________________________________________________
                                           \^o~_.
     .~.                           ______  /( __ )
     /V\         Toys story         \__  \/  (  V
   //   \\                            \__| (__=v
  /(     )\                        |\___/     )
    ^^-^^                           \_____(  )
     Tux                        Konqui     \__=v
 __________________________________________________