Re: [Scrollkeeper-devel] Handling relocatable packages

[Dan M. <d-...@uc...>]

On Thu, 23 Nov 2000, Nik Clayton wrote:

> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> > 
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
> 
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?

On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
gives me 506 packages which are non-relocateable, the last of which is
xbill:)

I think Red Hat would be better qualified at explaining this decision.

> > The current way things are designed is that it will have one contents list
> > for the system.  
> 
> I think that's a bad idea.
> 
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does.  I'm describing
> what I think SK should do, and what the interfaces should be.  It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
>
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed.  By specifying which OMF directories you want to include,
> > these databases could be made to differ.  We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned.  This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
> 
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.

I must have been unclear above.  We definitely want to keep all of the
original metadata in OMF files and seperate from the database.  These OMF
files should never be edited after they are initially installed (and
possibly initially fixed to have the right IDENTIFIER during the
post-install script, for relocatable packages).  This way we are
guaranteed to have an authoritative copy preserved in case the databases
are corrupted and to make upgrades simpler.

What I was trying to say above is that you can specify where the database
lives with a -p <path> flag.  If I run scrollkeeper-update -p /tmp/junk/
and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
all of the OMF files in the directory(ies) it is configured to read, and
then dumps a brand new database(ie. contents list, toc, index, ...) under
/tmp/junk/. This database won't generally be any different from the one in
its normal location (/var/lib/scrollkeeper/) unless it is running under a
different configuration.  This would happen for a user who has his own
configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
should be added to the database (typically under $HOME/.scrollkeeper/db/
perhaps).

> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization.  The problem is
> > that somebody has to specify what each database contains.  So it works for
> > users, because they can hand-tweak the contents.  It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager.  So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.  
> 
> I think it does, as long as a mechanism is provided so that sk can work 
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
> 
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF.  OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
> 
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.

We definitely want to support multiple databases, each of which has a
Contents List.

I am still uncertain if we want inheritance between contents lists though.

In many cases users will not want inheritance, because they lose any
control over what they inherit.  If instead they generate their own
database, they have complete control over all its parts, including those
parts which they would have otherwise inherited.  One way they could do
this is to over-ride one or more OMF files by keeping local
copies.  Another way is to have a local categories file which differs from
the system's. This approach gives the user complete control over their
database (including their contents lists) which would be much more
difficult and awkward to achieve by inheriting.

Now we can come up with instances where inheritance is useful.  If a user
wanted all the man pages in their help system, they might be better served
by inheriting the system man page database.  They would be limited though,
in that they could not specify which sections of the man pages they want.
This would be solved by having a database for each section of man
pages.  Then these sections could be inherited by users which would be
much faster than regenerating them for each user.

I think we really need to be very clear on how this will work and how it
will be used.  Would all databases be generated with the same categories
list?  If not, how do you handle merging them?  Presumably there is a
"default" categories list which gets enforced on the final contents list.

I think performance measurements would be useful here.  The example above
of using inherited databases with man pages has an advantage because
updating the database is faster.  It may save disk space, but this is
probably negligable. It has the downside of slowing down run-time
performance and making things substantially more complex and fragile.

> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document.  The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.

If I understood you, the "system" (ie nothing under /home) would have
multiple databases.  So when documentation is registered, it must be
determined which database to register it in.  Since this must be passed to
the registration script, it is determined by the packager based on
information about the content of the docs.  Thus, it is based on document
metadata.  (eg. this is a man page so it goes in the man page database,
this one is a KDE doc so it goes in the KDE database, ...)  This type of
metadata is stored in the OMF file already.

What various databases do you envision?

[SNIP]

> > What is the advantage of breaking the main database into pieces?
> 
> Flexibility for the future.  
> 
> > >   2.  Maintaining a TOC from the Contents List
> > > 
> > >   3.  Maintaining an Index from the Contents List
> > > 
> > >   4.  Maintaining a Categories list
> > 
> > Well, this is somewhat debatable and worthy of discussion.  Somebody has
> > to provide and maintain the categories list.  On one side, we can just
> > define it as part of OMF/sk and install a static list with sk.  Then no
> > maintenance is necessary.  Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
> > sys admin could drop in a replacement if they want.  For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
> 
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist.  I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.

We could provide an option to not use categories lists.  We definitely
should provide the ability to use them however, and we should specify a
suggested list.  

> However, that's not the same thing as saying
> 
>   If a categories file exists (in some, as yet undecided format), SK
>   should have the option of warning the SA that they're about to install
>   a document with a category that does not currently exist, and asking
>   whether they want to continue.  Naturally, the SA should have the
>   option of overriding this check with a command line flag.

We could provide a warning and even an interactive mode as an option.  

> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > >   application will have the responsibility for handling user queries of the
> > >   SK TOC and Index files ]
> > 
> > sk should export searching functionality. 
> 
> How?  libsk.so?  By having a program that accepts search parameters and
> returns XML with the results?  Some other mechanism?

We definitely want libsk.so.  I'm not sure if we would want a command-line
program for this.

> > > Activities (2) and (3) require a Contents List.  Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> > > maintain several different TOCs and Indices from several different Contents 
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> > 
> > For a given database, I agree.  Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one.  Is
> > this what you have in mind?  If you think we want more than one contents
> > list in the system's database, please explain why.
> 
> Flexibility and efficiency.  Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on.  That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.

It will be big.  Hopefully we can start running some tests on performance
some time soon to see how big of an issue this becomes.  It is not clear
that merging two medium sized lists into a big list is any better than
starting with a big list.  You still have a big list in the end, only you
save the part where you have to process and store two medium lists all the
time.

> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man.  So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.

This would speed things up a bit when one installs a new OMF file if man
pages were treated the same as other documents.  However processing of man
pages are done seperate from other applications since the OMF files for
most (all?) man pages are generated and can be stored in a special
location.  This way the man pages are updated with cron each night (which
must also make sure to create OMF metadata for each man page) and the
normal docs are registered whenever an application installs.  Because they
are processed at different times they do not slow each other down.

Aside from rebuilding the database in cases where it becomes corrupt, I'd
like this to be a zero maintenance system that works out of the box.  I
don't think we should focus on unusual things a SA may want to
do.  Instead we should make it work well and in a robust manner so the SA
won't be tempted to try to "fix" it.

> > > What sort of documentation can the Contents List going to point to?
> > > 
> > >   1.  Locally installed standard documentation that comes with the system,
> > >       such as man pages, or Info documents.
> > 
> > Perhaps.  I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below.  For man and info, we would have to update this information
> > off of cron unfortunately.  So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
> 
> I don't quite agree.  As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
> 
> First, the user tells SK about the documentation.
> 
> Second, the user assigns some OMF to the documentation.
> 
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.

Why do we need the former step?  The OMF data includes the IDENTIFIER,
which makes the first step unnecessary.

[SNIP]

> Note that I don't mean that SK goes and writes a new, master contents
> list file.  Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).

If we import a contents list it could only have documents on the Net on it
or the docs would generally be unreachable.  However this could be a very
useful feature.  You could imagine a given project publishing a database
and docs on the Net and then importing them into your local
system.  Without the ability to inherit contents list one would have to
install the metadata locally.  This would be possible and could even be
automated.  We would need to consider this carefully to determine which
solution is simpler and more robust.

> > > DD:  sk must have a run time option to allow the end user to select which
> > >      TOC to update.
> > 
> > The databases are stored together.  So, this should be specified at the
> > same time as the contents list part of the database.  I don't think I want
> > my contents list coming from one database and my toc from another.
> > 
> > (Or am I misunderstanding you?  I'm not sure what you mean by "update" a
> > TOC.)
> 
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC.  That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.

Yes.

> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.

Not if only one contents list/toc pair live in each database directory.

> Or are you saying that the ToC should be stored within the Contents List?

No.

[SNIP]

> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.

The only way scrollkeeper knows that two docs of different
versions/formats/languages are otherwise the same is using the extra OMF
element (which doesn't yet exist) which assigns them an ISBN-type code
which is the same.  Since they have different IDENTIFIERS (and other
metadata), sk knows they are different documents.

[SNIP]

> > Why do we specify the format on the command line?  The FORMAT is specified
> > in the OMF already.  The packager should just install and register the
> > metadata instead of putting it on the command line.
> 
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we 
> give it.
> 
> This separation is necessary to support documents that have no separate
> OMF.  Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.

Perhaps we agree on this after all and are just talking past each
other.  So for docs with OMF we have two steps:
1) update the IDENTIFIER to point to the right URL
2) register the OMF file with sk

Really, #1 is only necessary for local docs (as opposed to remote
ones) and for relocatable packages.

For docs without OMF, we do two steps:
1) generate the best OMF file we can given the doc
2) register the OMF file with sk

> > Here I assume (hope?;)  that you mean the database directory and not the
> > contents list.  If so, we already do this with the "-p" flag.
> 
> Right.

So we would have multiple databases, each with exactly one Contents List,
TOC directory, etc., right?

> I'd like to see a mechanism to specify it on the command line (apart
> from anything else, it's easier to script).  sk-build-omf, as you
> suggest above, seems like a good compromise.

:)

> > Right now we just number them, which guarantees a unique identifier.  
> 
> Two problems with this:
> 
>   1.  Probably won't scale efficiently as the Contents List gets larger.
> 
>   2.  Makes it impossible to merge two separate Contents Lists, from two
>       or more sites (say you're on a univesity campus, where the biology
>       department and the chemistry department both make available their
>       list of documentation in two different Scroll Keeper Contents Lists).

When sk merges seperate databases, it can easily mark them so that it
knows where the pieces came.  One way to do this would be to append a few
digit code to the beginning of each id which describes which database it
came from.

>       Your document browser is going to need a mechanism to uniquely
>       identify each document.  If there's a strong probability that the
>       document IDs are not unique then the browsers are each going to
>       have to reinvent their own mechanism for ensuring that the
>       documents are unique.
>
>      The documents URL might be appropriate, but it's a bit unwieldy.

The document URL is a very good unique identifier, and it is already being
passed to the help browser.

> You really need a mechanism that can (as near as possible) guarantee
> globally unique document IDs.

I'm pretty sure that only one document lives at a given URL ;)

> > >   As you can see, not a lot of information in there so far.  "lang" and
> > >   "encoding" could also be specified on the command line, and if not 
> > >   specified, some defaults are used.
> > > 
> > >   I think we probably also need a modification time attribute, but I haven't
> > >   shown that here.
> > 
> > The modification time is stored in a seperate file which keeps track of
> > which documents are in the database and is used for database maintenance,
> > updating, installing, uninstalling, etc.  The contents list does not care
> > about the modification time.
> 
> It probably should.  Apart from the issues I outline above, this then
> also lets you do things like "Show me the documents in order, most
> recently modified first".

This sort of filtering should be handled by sk, which knows the timestamps
even though they are not part of the contents list.  (The contents list is
there to sort the documents according to a given set of instructions - the
categories file. It is not the primary source for the metadata or
metametadata. We have a seperate metametadata file.)

> > >  ==========================================================================
> > >   Category: Unfiled
> > > 
> > >     The App-Name manual
> > > 
> > >       Include meta information from the OMF here (SK ID = MD5)
> > > 
> > >       Formats: Local HTML (split), PDF, Remote HTML (split)
> > >  =========================================================================
> > 
> > This may be useful for people who don't have a help-browser aside from
> > Netscape.  Although Netscape won't know how to display most docs.
> 
> True.  However, we should allow the Index builder flexibilit in deciding
> what documents are included in the Index.  This may include producing a
> special Index that only contains documents available in HTML, plain
> text, and PDF (for example).

What do you mean by "index"?  The definition I gave for "index" in prop#3
is what you find in the back of a book - an alphabetical list of concepts
and links to where they are discussed in the given work.  My impression is
that you mean the Contents List here.  Is that right?

I agree that we will want searching/filtering features in sk that allow
people to control its behavior as you suggest.

Dan