scrollkeeper-devel — Developer discussion list for ScrollKeeper

Re: [Scrollkeeper-devel] Handling relocatable packages

[Ali A. <ali...@au...>]

* Nik Clayton (ni...@no...) wrote at 19:17 on 23/11/00:
> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> > 
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
> 
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?

GNOME is non-relocatable...Nautilus is non-relocatable (I maybe incorrect, somebody
told me they were non-relocatable)

The reason is the "datadirs" are define'd at compiletime.

Lets take an example....I have an application called gLife that uses a glade
file for its UI. The glade file is installed in $(prefix)/share/glife (prefix
is defined at ./configure time)....so to load the glade file, I need the
/path/ to the glade file. This is defined during compilation as GLIFE_DATADIR. 

Now, if you compile the application into an RPM for '/usr' then you try
installing it in '/opt', the application will still look for the glade file in
/usr/share/glife.

Now, I know there are probably hacks that can be done to get around this. I
know that this may not be "theoretically" correct, etc. etc. etc. But the fact
is there /ARE/ non-relocatable packages out there, and scrollkeeper should
support both types of packages.
 
> > The current way things are designed is that it will have one contents list
> > for the system.  
> 
> I think that's a bad idea.
> 
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does.  I'm describing
> what I think SK should do, and what the interfaces should be.  It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
> 
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed.  By specifying which OMF directories you want to include,
> > these databases could be made to differ.  We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned.  This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
> 
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.

I agree, they should be separate.
 
> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization.  The problem is
> > that somebody has to specify what each database contains.  So it works for
> > users, because they can hand-tweak the contents.  It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager.  So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.  
> 
> I think it does, as long as a mechanism is provided so that sk can work 
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
> 
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF.  OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
> 
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.
> 
> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document.  The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.
> 
> Also, consider a regular user who occasionally installs applications and
> documentation in to $HOME.
> 
> It's much easier for them to add a flag to a command line (or, better
> still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
> than it is for them to edit OMF files before installing them.
> 
> > What is the advantage of breaking the main database into pieces?
> 
> Flexibility for the future.  

There should be support in mind for "multiple" databases in my opinion. This
way, a person can set an environment flag, or a command-line option and be
able to load an entirely new database (along with his old one). 

Perhaps you should have a '/etc/scrollkeeper.conf' file that has a list of
various databases as well as various configuration options. 

In addition to this, I think scrollkeeper should handle any databases found in
'~/.scrollkeeper/'

So, multiple databases does make sense in my opinion (think of it this way,
just because YOU don't need it, or can't find a reason to use it, it doesn't
mean that it is not useful for others)..

From a desktop perspective it might not be necessary to have multiple
databases, but from a "sysadmin" perspective it does.
 
> > >   2.  Maintaining a TOC from the Contents List
> > > 
> > >   3.  Maintaining an Index from the Contents List
> > > 
> > >   4.  Maintaining a Categories list
> > 
> > Well, this is somewhat debatable and worthy of discussion.  Somebody has
> > to provide and maintain the categories list.  On one side, we can just
> > define it as part of OMF/sk and install a static list with sk.  Then no
> > maintenance is necessary.  Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
> > sys admin could drop in a replacement if they want.  For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
> 
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist.  I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.

FAQ/Manual/HOWTO are not category types in your opinion...What I mean by
categories is more like "subject" - i.e. "Audio/MP3 Players' or
'Office/Spreadsheet' or 'X11/Internet/FTP Clients' etc.

The fact that it is a FAQ/HOWTO/Manual should be reflected in the name of the
document (I think this is the "purpose" of a document).

There should also be another field that specifies the type of document (HTML,
Info, Man, DocBook, plain text, etc.)

> However, that's not the same thing as saying
> 
>   If a categories file exists (in some, as yet undecided format), SK
>   should have the option of warning the SA that they're about to install
>   a document with a category that does not currently exist, and asking
>   whether they want to continue.  Naturally, the SA should have the
>   option of overriding this check with a command line flag.

I am saying just a HTML file, or plain text file just LISTING the categories -
nothing fancy...Scrollkeeper would have no knowledge of this file and will not
interact with it in anyway.

The scrips should NEVER EVER become "interactive". Just use whatever is in the
category (in the OMF file).

> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > >   application will have the responsibility for handling user queries of the
> > >   SK TOC and Index files ]
> > 
> > sk should export searching functionality. 
> 
> How?  libsk.so?  By having a program that accepts search parameters and
> returns XML with the results?  Some other mechanism?

I prefer a library that returns the xmlDocPtr of the file...you can then parse
it as you wish. Note: with a library you could _EASILY_ create a command-line
application that would just "dump" the XML tree to stdout (if you need it)

> > > Activities (2) and (3) require a Contents List.  Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> > > maintain several different TOCs and Indices from several different Contents 
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> > 
> > For a given database, I agree.  Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one.  Is
> > this what you have in mind?  If you think we want more than one contents
> > list in the system's database, please explain why.
> 
> Flexibility and efficiency.  Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on.  That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.
> 
> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man.  So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.

I think this "optimization" is unncessary at this time. We have no data _AT
ALL_ on the efficiency in scrollkeeper. We don't know if it will be slow or
fast, so splitting up databases for this reason is unacceptable in my opinion.

On the other hand, I do think we should support multiple databases (through a
command-line option, and possible an environment variable (and API functions
when the library is done))

> > > What sort of documentation can the Contents List going to point to?
> > > 
> > >   1.  Locally installed standard documentation that comes with the system,
> > >       such as man pages, or Info documents.
> > 
> > Perhaps.  I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below.  For man and info, we would have to update this information
> > off of cron unfortunately.  So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
> 
> I don't quite agree.  As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
> 
> First, the user tells SK about the documentation.
> 
> Second, the user assigns some OMF to the documentation.
> 
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.
> 
> Consider the man page, for example.  If I want to tell SK about all my
> section 1 man pages, I'd like to be able to do something like
> 
>     cd /usr/share/man/man1
>     sk-install-doc *.1
> 
> which would tell SK about all my section 1 manual pages.  Actually, what
> I'd probably do is something like
> 
>     sk-install-doc -category "Man Pages:Section 1" *.1
> 
> to assign a category to them.  At this point, there's no OMF for them,
> but SK can synthesise the bare bones from the filename and the category.
> 
> Suppose, then, that I want to add some OMF for ls.1.  I might do
> something like this;
> 
>     sk-extract-omf ls.1 > ls.omf
>     vi ls.omf
>     [edit edit edit]
>     sk-install-omf ls.1 ls.omf
> 
> which would pull out the OMF that SK synthesised for me in the earlier
> step (presumably as XML).  I can then edit it, and then install the
> edited OMF back in to SK's control.
> 
> I think it's highly likely that *roff -man (and in particular, *roff
> -mdoc) is going to be easy to extract OMF from.  So what we might end up
> with is a command to extract OMF from -man formatted documents.  It
> might work like this;
> 
>     cd /usr/share/man/man1
>     foreach man in (*.1)
>         extract-omf -source man $man > `basename $man .1`.omf
>     end
> 
> which would iterate over *.1, extracting the OMF and writing it to
> *.omf.  With that done, the end user might then be able to do
> 
>     foreach man in (*.1)
>         sk-install-doc -doc $man -omf `basename $man .1`.omf
>     end
> 
> which could install the document, and the pregenerated OMF, into SK with
> one command.
> 
> I hope the design ethos that's coming across from all of this is to have
> lots of small tools, each of which does one thing.

I disagree here...Without the OMF data ScrollKeeper shouldn't do jack shit to
the document. It shouldn't even know the document exists.

If somebody wishes to write an application that would take a document, create
an OMF file from the document, and register it with ScrollKeeper - well, that
would be nice. But its not /ScrollKeepers/ responsibility.

Maybe something like this can be done in the future, but I don't see this as
an 'essential feature'...

Also it would be "nice" to have a command-line tool that would allow you
modify the OMF metadata based on the command-line options.

> > >   2.  Locally installed system documentation, installed by the SA team.
> > 
> > SA =? sys admin
> 
> Yes.
> 
> > >   3.  Locally installed documentation, installed by the user.
> > 
> > Yes.  Although like much of this discussion, this feature will come in a
> > post-0.1 release.
> 
> As I say, I'm not thinking in terms of 'release' yet.  I'd like to get
> some specific functionality nailed down and agreed on.
> 
> [ Incidentally, and talking about the implementation -- one of the
>   things that occured to me last night; sk could have a command
>   structure like that of cvs, where instead of having a bunch of
>   sk-foo-bar commands, you have one 'sk' command, that takes options,
>   and then subcommands, which also take options.
> 
>       sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
>       sk install-omf /path/to/omf
> 
>   and so on. ]

Difficult I think since the SK stuff are two distinct separate tools :) but it
would be an interesting thing to do...

> > > I think it is fair to say that, when run, sk will need to construct it's
> > > global Contents List by parsing multiple, smaller, contents lists.  In much
> > > the same way that SGML catalog files can include one another at the moment.
> > 
> > Why? It sounds very nice and wonderful.  However, this adds a layer of
> > complexity without any obvious benefits.
> 
> As above.
> 
> Note that I don't mean that SK goes and writes a new, master contents
> list file.  Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).

You could just re-construct one large XML tree in memory from multiple smaller
XML files. You would then conduct all operations on this in-memory XML tree.

It won't make much diff. performance wise if you do this on one LARGE xml
file, or multiple smaller ones i believe (provided you are doing the above
approach of building a large XML tree from smaller ones).

> > > DD:  sk must have a run time option to allow the end user to select which
> > >      TOC to update.
> > 
> > The databases are stored together.  So, this should be specified at the
> > same time as the contents list part of the database.  I don't think I want
> > my contents list coming from one database and my toc from another.
> > 
> > (Or am I misunderstanding you?  I'm not sure what you mean by "update" a
> > TOC.)
> 
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC.  That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.
> 
> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.
> 
> Or are you saying that the ToC should be stored within the Contents List?
> 
> > > DD:  sk must have a run time option to allow the end user to select which
> > >      Index to update.
> > 
> > (as above)
> 
> Ditto :-)
> 
> > > DD:  One of the sk commands must be to generate a usable Contents List 
> > >      (which really means {X}HTML version, to start with) given a Contents
> > >      List which is in sk internal format.
> > 
> > Everything is XML right now, so this is not an issue.  If we decide
> > another internal format is better we should still export XML to the
> > browser.  As for HTML, this is possible and may be a nice interface for
> > people who want to create a very minimalistic web browser.
> 
> All the world is not yet XML, and not everyone runs KDE or Gnome.  Being
> able to convert the ToCs and Indices to other formats is vital.
> Fortunately, since the ToC and Indices are XML, this should be easy,
> using something like XSLT, so we can probably just consider this
> functionality that we get for free.

Well, even if you don't use KDE or GNOME, you would still need to parse
ScrollKeeper's file-format...

Lets say we /don't/ use XML, you would still have to handle whatever internal
data structure is used by ScrollKeeper. We just picked XML because it is
(relatively) easier to parse.
 
> > > DD:  It is possible for a document to change, but for the OMF information
> > >      to be neglected.  sk should maintain a modification time stamp for
> > >      each document and its corresponding OMF, to allow the end user to 
> > >      determine when that might have happened, and to correct for it.
> > 
> > For the current version of sk, which does not yet allow for remote
> > documents, only the OMF file's timestamp is watched.  This is because if
> > somebody edits the doc and not the OMF, then it is their fault for not
> > updating the OMF, not sk's.  In reality whenever you install a new version
> > of a package, even if the OMF is not updated (as it should), it still
> > updates the old version so it gets a new timestamp and the new doc gets
> > registered despite any negligence on the part of the maintainer to update
> > the OMF (or the doc for that matter;).
> > 
> > This mainly becomes an issue for:
> > 1) remote docs
> > 2) docs which users write and modify themselves.
> > 
> > So we may indeed need to start tracking the doc timestamps when we support
> > these features.  We have to be careful about constantly checking the
> > timestamps on a bunch of docs which are spread around the internet
> > though.  There will be many cases where this task cannot complete, or else
> > takes an unacceptable amount of time.
> 
> This should be configurable behaviour, something like;
> 
>     sk --check-remote-timestamps --remote-timeout=30
> 
> Actually, this implies another DD
> 
> DD:  sk must track the validity of the OMF in relation to it's document.
>      OMF might be:
> 
>         valid      This OMF was installed with the document, and neither
> 	           the document or the OMF's timestamps have changed.
> 
>         invalid    The document and OMF have different timestamps
> 
> 	unknown    The document and OMF have the same timestamps, but
> 	           the most recent attempt to confirm this (particularly
> 		   for remote documents) failed.
> 
>      sk should be able to report on documents who have OMF which is not
>      valid, so the SA can investigate further.
> 
> > > DD:  One document, multiple output formats.  The user might have installed
> > >      the same document three times, once as HTML, once as PDF, and once as
> > >      plain text.  The three installations are the same document, and have
> > >      the same OMF associated with them.
> > 
> > No. These definitely do not have the same OMF, unless they are identical
> > copies.  The OMF file specifies the document format in FORMAT.DTD and
> > FORMAT.MIME.  The docs also have different IDENTIFIER's.
> > 
> > (We may decide to expand this a little bit.  FORMAT.STYLESHEET may be
> > nice, as it would allow the doc authors to dictate the appearance of their
> > doc.  So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> > icon, etc.)
> > 
> > >      sk should treat this as one document with three formats, rather than
> > >      three separate documents.     
> > 
> > If you look in the ldp-meta archives from several months ago, I proposed
> > we add a document id to the OMF.  This would allow us to identify if two
> > documents are "The-Same".  I call two documents "The-Same" if they are the
> > same document only different versions, languages, formats, etc.  If two
> > documents have the same title, they are not necessarily
> > "The-Same".  Likewise, if I change the title of my document at some point
> > in time, I can have two docs which are "The-Same" but not having the same
> > title.
> > 
> > Obviously this sort of information will be very nice to have.  You can do
> > things like "Get me this document in language xxx." or "Show me the latest
> > version of this document on the Net."  or "Show me the previous version of
> > this document from the Net."
> > 
> > >      [ This is probably going to be the most contentious piece of this 
> > >        writeup, and it might change.  In particular, it also suggests that
> > >        translations of a document should be treated as the same document,
> > >        just in a different language, and I'm not sure that's a good idea.
> > >        Then again, it might be. ]
> > 
> > I guess we were thinking along similar lines :)
> 
> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.

Translations are considered to be "separate" documents for me.

I don't know about different formats...that is a difficult one - but why would
we need to distinguish between instances of a document based on format?

I like the idea of an "id" of a document, because then you can have diff.
versions of a doc. It is difficult to determine wether translations should
have their own id...I recommend you just flip a coin.

The good thing about an id (unique id's can be obtained using 'uuid-gen') is
that you can query any server in the globe for that particular id and it will
retain that document (provided you also provide the version number, and
possible the format)
 
> > > There are a number of tasks that the sk end user is going to want to tell 
> > > sk to do.  They are;
> > > 
> > > Install new documentation that comes bundled with OMF
> > > 
> > >   The user has just downloaded and installed a package that comes with one
> > >   or more bundled documents, and the package maintainer has thoughtfully
> > >   provided an OMF file for each document.
> > > 
> > >   Each document has been provided in HTML and PDF format.
> > > 
> > >   Suppose the user installs the HTML documentation first, and installs it
> > >   it into /usr/local/share/doc/app-name/html/, where normal file is called
> > >   index.html.
> > > 
> > >   [ Everywhere I say "the user runs" I really mean "The package maintainer
> > >     does this by means of a post-install script that they've already written.
> > >     It is assumed that this post-install script knows the directories that
> > >     documentation has been installed in, by virtue of being part of the 
> > >     package that did the installation in the first place. ]
> > > 
> > >   The user runs 
> > > 
> > >      sk-install-doc -format html-split -db /var/db/sk/master.xml \
> > >          /usr/local/share/doc/app-name/html/index.html
> > 
> > Why do we specify the format on the command line?  The FORMAT is specified
> > in the OMF already.  The packager should just install and register the
> > metadata instead of putting it on the command line.
> 
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we 
> give it.
>
> This separation is necessary to support documents that have no separate
> OMF.  Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.

ScrollKeeper is here because (and for) Document metadata - if it doesn't
exist, then ScrollKeeper should not know about the file.


[SNIP]

Too be honest, I got tired of reading this very very very long mail (it's been
one hour!!!!)

Anyway - I think somebody needs to come up with SPECIFIC suggestions, and
split this really long thread/mail into separate smaller, more manageable
threads.

Regards,
Ali 

Re: [Scrollkeeper-devel] Handling relocatable packages

[Ali A. <ali...@au...>]

* Laszlo Kovacs (las...@su...) wrote at 20:07 on 23/11/00:
> Guys,
> 
> You have to cut back on the email size, otherwise nobody is going to
> contribute to this discussion.

AGREED!!!!!!!!!

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Ali A. <ali...@au...>]

* Dan Mueth (d-...@uc...) wrote at 01:48 on 23/11/00:
> 
> On Wed, 22 Nov 2000, Ali Abdin wrote:
> 
> > * Dan Mueth (d-...@uc...) wrote at 22:57 on 22/11/00:
> > > 
> > > On Wed, 22 Nov 2000, Ali Abdin wrote:
> > > 
> > > > On Tue, 21 Nov 2000, Bob Stayton wrote:
> > > 
> > > > > 1.  A predetermined complete categorization of content is
> > > > > not useful or even possible.  Rather, an open framework
> > > > > of general categories can be established so providers
> > > > > have a place to hang their stuff.  But providers need
> > > > > the ability to add subcategories.
> > > > 
> > > > As far as I know - anyone can put their document in any category they 
> > > > like. But there should be a set of "recommended" categories for people to 
> > > > use.
> > > > 
> > > > This happens in RPM, people can put their category as anything in their 
> > > > Spec file, but usually they pick one of the "standard" ones.
> > > > 
> > > > Scrollkeeper will not "enforce" a strict set of categories, but I think a 
> > > > recommended list of categories should be published
> > > 
> > > I agree with this point of view.  Scrollkeeper itself is just a software
> > > package, not a specification.  The OMF is the spec for the metadata.  We
> > > will develop another spec for a standard category system for the contents
> > > list.  However, people should be able to use scrollkeeper with their own
> > > contents list.  (eg. in a classroom or computer lab for newbies)  I'd like
> > > to just point scrollkeeper to whichever categorization file it should use.
> > > 
> > > We need to decide what happens to docs which fall out of the
> > > categorization.  We have 3 options:
> > > 1) they are omitted
> > > 2) they are put in a generic "other" category(ies)
> > > 3) they are put in regardless of the categories file
> > > 
> > > We may want a configuration parameter to determine which of these 3 are
> > > done, since depending on your situation you may want any of these.
> > 
> > I disagree. ScrollKeeper should not "point" to or know about any categories.
> > It just uses whatever is specificed in the OMF file. So if the category is
> > 'Audio/MP3 Player' it gets stays that, and if the category is 'ABC/XYZ' it stays
> > that 'way'.
> > 
> > What ScrollKeeper should do is provide a "list" (on the webpage, in the docs
> > section, etc.) of "recommended" categories to use.
> 
> Yes.  Nothing I said contradicts this.  I'm just saying that in addition
> to publishing a "recommended" categorization system, we also accomodate
> documents which fall out of this recommended categorization and
> categorizations other than the recommended one.

Well - you actually said "We need to decie what happens to docs which fall out
of the categorgization" - I am saying, you do NOTHING when they are not in the
categorization tree. You just use the provided category "as-is"
 
> So if a user wants to copy the standard contents list and add a section
> called "DocsIWrote" or "Favorites" to it, they can do this.  Similarly,
> they can (eventually) create OMF files under $HOME/.scrollkeeper/omf/
> which can populate these sections they create in their contents list tree.

You don't need a standard contents list as far as I know?
Why would you need it if we "don't care" about the category?

> [SNIP]
> 
> > > It would still be nice to have these example programs, or even have a
> > > complete help browser which does not depend on all the GNOME and KDE
> > > libraries.  I don't personally place this high on my priorities list
> > > though.  If anybody else wants to write a desktop independent browser
> > > which uses ScrollKeeper then this would certainly be a Good Thing.
> > 
> > I have a gripe with Scrollkeeper right now...It appears that the only way to
> > interface with it is to use applications/scripts. 
> > 
> > I thought we were going to have a library that implements this sort of
> > functionality (to get the contents list, etc.) - I really don't like the idea
> > of using pipe's for the help browser (a library would also be more
> > efficient)...
> 
> Yes. This is the plan.  We have not worked out the details here, so if you
> would like to suggest how this API would work, please do.

Actually - the way it usually works is you provide the API for the
functionality you /think/ we need, we comment on it, you revise, then we start
using it, then we start complaining and you revise again ;)

Basically, the return time for most operations would be a xmlDocPtr that the
application/viewer would have to parse ;) Thats the only input I have at this
time (its 2AM and I'm tired)

> 
> > Another mini-gripe...In scrollkeeper/cl/README it says 'foo.xml is the
> > document metadata file' - I thought we had agreed to use 'foo.omf' instead to
> > help identify the file easier?
> 
> Yes.  We should go through and make sure this is fixed everywhere.  I see
> that the example package does it correctly.
> 
> > Another question...Where will be the "burden" be of searching the index? For
> > example, say I want to "filter" the help files on the category 'Internet/Browsers'...Is
> > this the responsibility of the help browser, or is it in Scrollkeeper's
> > domain?
> 
> Most metadata searches belong in ScrollKeeper.  This is partly because
> ScrollKeeper should be handling the metadata instead of the help browser,
> partly to reduce the work which must be duplicated in each help browser, 
> and partly because we will eventually have ScrollKeeper acting as a
> "middle-man" between the help browser and metadata servers on the Net
> which can handle these queries.
> 
> The particular example you give is more of filtering (as you called
> it) then searching, so it isn't as obvious where this belongs.  I think it
> belongs in ScrollKeeper as well.
> 
> Is this something we are likely to use any time soon?  I think at least
> for Nautilus we will not need this for some time.  If anybody else wants
> it, then we can move it up on the TODO list.

Well, I'm not sure Nautilus will be moving to ScrollKeeper quite so soon (i.e.
if you're gonna switch to a library, then why should we use the current set-up
that Sun submitted)...I'm not sure...we'll have to see.

> > The ways to implement the above are:
> > 
> > 1) We have an app/script that lists the entire contents, which you have to
> > pipe and then search on, then present
> > 2) We have an app/script that does the filtering for you, which you have to
> > pipe and just present
> > 3) We pass the xmlDocPtr of the entire XML database, which you have to search
> > on, and then present
> > 4) We pass the xmlDocPtr of the "filtered" XML database
> > 
> > I think its pretty obvious that I prefer #4 ;)
> 
> Sounds fine with me.  
> 
> It sounds like you have been thinking about the library ScrollKeeper
> should export to help browsers.  Perhaps you could write up a proposal
> describing what you think it should look like?

Actually, I'm way too busy to do that (to compound the matter, I have the
final round of midterms, then finals, then travelling to Kuwait) - I also have
a bunch of stuff that needs higher priority (backlog of patches for Nautilus
help stuff)...

if you can come up with something, great. If not, then I guess i'll try to do
it in my freetime...

Regards,
Ali

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

Guys,

You have to cut back on the email size, otherwise nobody is going to
contribute to this discussion.

Laszlo

> 
> On Thu, 23 Nov 2000, Nik Clayton wrote:
> 
> > On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > > There should be no such thing as a non-relocatable package.
> > >
> > > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > > should support them both in the simplest possible way.
> >
> > Can you give me an example of non-relocatable package, and it's
> > justification for being non-relocatable?
> 
> On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> gives me 506 packages which are non-relocateable, the last of which is
> xbill:)
> 
> I think Red Hat would be better qualified at explaining this decision.
> 
> > > The current way things are designed is that it will have one contents list
> > > for the system.
> >
> > I think that's a bad idea.
> >
> > Also, I should be clear here -- I'm not talking about how the system is
> > currently designed, or what the code currently does.  I'm describing
> > what I think SK should do, and what the interfaces should be.  It's kind
> > of hard to evaluate the existing functionality at the moment, because I
> > haven't seen a roadmap that lists explicit design decisions, and things
> > that will and won't be supported.
> >
> > > Of course you can always create another one somewhere
> > > else using the -p flag, but generally they would contain the same
> > > information and only the default one will get updated each time a document
> > > is installed.  By specifying which OMF directories you want to include,
> > > these databases could be made to differ.  We don't currently have a
> > > mechanism for supporting or specifying multiple OMF directories, but that
> > > would be fairly straightforward and is planned.  This will make it very
> > > simple for users to have their own local databases which vary from the
> > > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > > which add to or over-ride the system defaults.
> >
> > This now sounds different from design doc #3, which, IIRC, suggested
> > that the contents list, and the OMF information, would be kept separate.
> 
> I must have been unclear above.  We definitely want to keep all of the
> original metadata in OMF files and seperate from the database.  These OMF
> files should never be edited after they are initially installed (and
> possibly initially fixed to have the right IDENTIFIER during the
> post-install script, for relocatable packages).  This way we are
> guaranteed to have an authoritative copy preserved in case the databases
> are corrupted and to make upgrades simpler.
> 
> What I was trying to say above is that you can specify where the database
> lives with a -p <path> flag.  If I run scrollkeeper-update -p /tmp/junk/
> and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
> all of the OMF files in the directory(ies) it is configured to read, and
> then dumps a brand new database(ie. contents list, toc, index, ...) under
> /tmp/junk/. This database won't generally be any different from the one in
> its normal location (/var/lib/scrollkeeper/) unless it is running under a
> different configuration.  This would happen for a user who has his own
> configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
> should be added to the database (typically under $HOME/.scrollkeeper/db/
> perhaps).
> 
> > > It is not clear how and why one would want to use multiple contents lists
> > > except for individual users and/or hand customization.  The problem is
> > > that somebody has to specify what each database contains.  So it works for
> > > users, because they can hand-tweak the contents.  It doesn't work very
> > > well for system-wide databases, since everything must be specified by the
> > > packager.  So while you could imagine GNOME documents being kept in one
> > > list and KDE documents being kept in another list, this doesn't really
> > > achieve anything.
> >
> > I think it does, as long as a mechanism is provided so that sk can work
> > out where to put the information if the user doesn't indicate otherwise.
> > A config file that specifies the default Contents List should suffice.
> >
> > > The information which would be passed on the command
> > > line to sk to indicate which list the doc goes in is really metadata which
> > > belongs in the OMF.  OMF and XML provide everything we need to have one
> > > database and understand the scope of each piece. (eg. whether it is a
> > > GNOME doc or KDE doc.)
> >
> > If you support multiple Contents Lists (and you have to, just to support
> > end users installing docs and not having write access to the system wide
> > Contents List) then you need to be able to specify which Contents List
> > to use.
> 
> We definitely want to support multiple databases, each of which has a
> Contents List.
> 
> I am still uncertain if we want inheritance between contents lists though.
> 
> In many cases users will not want inheritance, because they lose any
> control over what they inherit.  If instead they generate their own
> database, they have complete control over all its parts, including those
> parts which they would have otherwise inherited.  One way they could do
> this is to over-ride one or more OMF files by keeping local
> copies.  Another way is to have a local categories file which differs from
> the system's. This approach gives the user complete control over their
> database (including their contents lists) which would be much more
> difficult and awkward to achieve by inheriting.
> 
> Now we can come up with instances where inheritance is useful.  If a user
> wanted all the man pages in their help system, they might be better served
> by inheriting the system man page database.  They would be limited though,
> in that they could not specify which sections of the man pages they want.
> This would be solved by having a database for each section of man
> pages.  Then these sections could be inherited by users which would be
> much faster than regenerating them for each user.
> 
> I think we really need to be very clear on how this will work and how it
> will be used.  Would all databases be generated with the same categories
> list?  If not, how do you handle merging them?  Presumably there is a
> "default" categories list which gets enforced on the final contents list.
> 
> I think performance measurements would be useful here.  The example above
> of using inherited databases with man pages has an advantage because
> updating the database is faster.  It may save disk space, but this is
> probably negligable. It has the downside of slowing down run-time
> performance and making things substantially more complex and fragile.
> 
> > Yes, you could put that information in the OMF, but, as I understand it,
> > the OMF contains meta information about the document.  The location of
> > the Contents List is not meta information about the document, so should
> > not be in the OMF.
> 
> If I understood you, the "system" (ie nothing under /home) would have
> multiple databases.  So when documentation is registered, it must be
> determined which database to register it in.  Since this must be passed to
> the registration script, it is determined by the packager based on
> information about the content of the docs.  Thus, it is based on document
> metadata.  (eg. this is a man page so it goes in the man page database,
> this one is a KDE doc so it goes in the KDE database, ...)  This type of
> metadata is stored in the OMF file already.
> 
> What various databases do you envision?
> 
> [SNIP]
> 
> > > What is the advantage of breaking the main database into pieces?
> >
> > Flexibility for the future.
> >
> > > >   2.  Maintaining a TOC from the Contents List
> > > >
> > > >   3.  Maintaining an Index from the Contents List
> > > >
> > > >   4.  Maintaining a Categories list
> > >
> > > Well, this is somewhat debatable and worthy of discussion.  Somebody has
> > > to provide and maintain the categories list.  On one side, we can just
> > > define it as part of OMF/sk and install a static list with sk.  Then no
> > > maintenance is necessary.  Or, we may consider having this specified by a
> > > configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
> > > sys admin could drop in a replacement if they want.  For example you could
> > > chop out all the HOWTO's and man pages if you know newbies who never touch
> > > a command line will be using your machine.
> >
> > I don't think SK should mandate the existence of a categories list, or
> > mandate that certain categories should exist.  I expect that some well
> > defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> > "Gnome") will spring up.
> 
> We could provide an option to not use categories lists.  We definitely
> should provide the ability to use them however, and we should specify a
> suggested list.
> 
> > However, that's not the same thing as saying
> >
> >   If a categories file exists (in some, as yet undecided format), SK
> >   should have the option of warning the SA that they're about to install
> >   a document with a category that does not currently exist, and asking
> >   whether they want to continue.  Naturally, the SA should have the
> >   option of overriding this check with a command line flag.
> 
> We could provide a warning and even an interactive mode as an option.
> 
> > > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > >   application will have the responsibility for handling user queries of the
> > > >   SK TOC and Index files ]
> > >
> > > sk should export searching functionality.
> >
> > How?  libsk.so?  By having a program that accepts search parameters and
> > returns XML with the results?  Some other mechanism?
> 
> We definitely want libsk.so.  I'm not sure if we would want a command-line
> program for this.
> 
> > > > Activities (2) and (3) require a Contents List.  Note that a don't say "the
> > > > {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to
> > > > maintain several different TOCs and Indices from several different Contents
> > > > Lists on the same host (for example, a regular user might want to run sk
> > > > to maintain their own index and TOC of ~/my-docs).
> > >
> > > For a given database, I agree.  Eventually, I see the system having one
> > > database (with one contents list) and possibly each user having one.  Is
> > > this what you have in mind?  If you think we want more than one contents
> > > list in the system's database, please explain why.
> >
> > Flexibility and efficiency.  Suppose you have one big contents list that
> > contains information about all the documentation that's installed on the
> > system -- man pages, readmes, faqs, and so on.  That's going to be a
> > very big contents list, particularly when apps like Gnome and KDE get in
> > on the act.
> 
> It will be big.  Hopefully we can start running some tests on performance
> some time soon to see how big of an issue this becomes.  It is not clear
> that merging two medium sized lists into a big list is any better than
> starting with a big list.  You still have a big list in the end, only you
> save the part where you have to process and store two medium lists all the
> time.
> 
> > For example, on a BSD system, the contents of /usr/share/man change once
> > in a blue moon, as third party manual pages generally live under
> > /usr/local/share/man.  So the admin might want to create a Contents List
> > for /usr/share/man once, and keep it separate, because the SA knows its
> > never going to change, so it doesn't need to be included in any
> > processes SK undertakes to ensure that the Contents List is sync with
> > the filesystem.
> 
> This would speed things up a bit when one installs a new OMF file if man
> pages were treated the same as other documents.  However processing of man
> pages are done seperate from other applications since the OMF files for
> most (all?) man pages are generated and can be stored in a special
> location.  This way the man pages are updated with cron each night (which
> must also make sure to create OMF metadata for each man page) and the
> normal docs are registered whenever an application installs.  Because they
> are processed at different times they do not slow each other down.
> 
> Aside from rebuilding the database in cases where it becomes corrupt, I'd
> like this to be a zero maintenance system that works out of the box.  I
> don't think we should focus on unusual things a SA may want to
> do.  Instead we should make it work well and in a robust manner so the SA
> won't be tempted to try to "fix" it.
> 
> > > > What sort of documentation can the Contents List going to point to?
> > > >
> > > >   1.  Locally installed standard documentation that comes with the system,
> > > >       such as man pages, or Info documents.
> > >
> > > Perhaps.  I think we probably do want to include this information, using
> > > automagically generated (and therefore incomplete) OMF metadata as you
> > > suggest below.  For man and info, we would have to update this information
> > > off of cron unfortunately.  So there will be an unavoidable delay between
> > > when a document is installed and when it is registered with sk.
> >
> > I don't quite agree.  As I say below, I think installing a new document
> > should be a two step process, at least conceptually (as it could be
> > carried out by one command).
> >
> > First, the user tells SK about the documentation.
> >
> > Second, the user assigns some OMF to the documentation.
> >
> > This is what I'm getting at with the sk-install-doc and sk-install-omf
> > command separation.
> 
> Why do we need the former step?  The OMF data includes the IDENTIFIER,
> which makes the first step unnecessary.
> 
> [SNIP]
> 
> > Note that I don't mean that SK goes and writes a new, master contents
> > list file.  Just that, in memory, SK is capable of aggregating
> > information from multiple Contents Lists (which should be accessible by
> > URL, so that remote Contents Lists are feasible).
> 
> If we import a contents list it could only have documents on the Net on it
> or the docs would generally be unreachable.  However this could be a very
> useful feature.  You could imagine a given project publishing a database
> and docs on the Net and then importing them into your local
> system.  Without the ability to inherit contents list one would have to
> install the metadata locally.  This would be possible and could even be
> automated.  We would need to consider this carefully to determine which
> solution is simpler and more robust.
> 
> > > > DD:  sk must have a run time option to allow the end user to select which
> > > >      TOC to update.
> > >
> > > The databases are stored together.  So, this should be specified at the
> > > same time as the contents list part of the database.  I don't think I want
> > > my contents list coming from one database and my toc from another.
> > >
> > > (Or am I misunderstanding you?  I'm not sure what you mean by "update" a
> > > TOC.)
> >
> > It's my understanding that there's a 1:1 mapping between a Contents List
> > and a ToC.  That is, you can't have a ToC that includes information
> > about documents that aren't in the Contents List.
> 
> Yes.
> 
> > So if you have multiple Contents Lists, you need to have multiple ToCs
> > (and Indexes), and a way of saying that >this< Contents List was used to
> > produce >This< ToC.
> 
> Not if only one contents list/toc pair live in each database directory.
> 
> > Or are you saying that the ToC should be stored within the Contents List?
> 
> No.
> 
> [SNIP]
> 
> > Do you agree that making the distinction between a document, and one or
> > more instances of that document (different formats, different languages)
> > is worthwhile.
> 
> The only way scrollkeeper knows that two docs of different
> versions/formats/languages are otherwise the same is using the extra OMF
> element (which doesn't yet exist) which assigns them an ISBN-type code
> which is the same.  Since they have different IDENTIFIERS (and other
> metadata), sk knows they are different documents.
> 
> [SNIP]
> 
> > > Why do we specify the format on the command line?  The FORMAT is specified
> > > in the OMF already.  The packager should just install and register the
> > > metadata instead of putting it on the command line.
> >
> > Because at this point, all we've done is install a document with no OMF.
> > SK has to synthesise some OMF, based on any additional information we
> > give it.
> >
> > This separation is necessary to support documents that have no separate
> > OMF.  Adding "-format" to the command line is probably going to be
> > easier for an end user than having to create a separate file to return
> > to.
> 
> Perhaps we agree on this after all and are just talking past each
> other.  So for docs with OMF we have two steps:
> 1) update the IDENTIFIER to point to the right URL
> 2) register the OMF file with sk
> 
> Really, #1 is only necessary for local docs (as opposed to remote
> ones) and for relocatable packages.
> 
> For docs without OMF, we do two steps:
> 1) generate the best OMF file we can given the doc
> 2) register the OMF file with sk
> 
> > > Here I assume (hope?;)  that you mean the database directory and not the
> > > contents list.  If so, we already do this with the "-p" flag.
> >
> > Right.
> 
> So we would have multiple databases, each with exactly one Contents List,
> TOC directory, etc., right?
> 
> > I'd like to see a mechanism to specify it on the command line (apart
> > from anything else, it's easier to script).  sk-build-omf, as you
> > suggest above, seems like a good compromise.
> 
> :)
> 
> > > Right now we just number them, which guarantees a unique identifier.
> >
> > Two problems with this:
> >
> >   1.  Probably won't scale efficiently as the Contents List gets larger.
> >
> >   2.  Makes it impossible to merge two separate Contents Lists, from two
> >       or more sites (say you're on a univesity campus, where the biology
> >       department and the chemistry department both make available their
> >       list of documentation in two different Scroll Keeper Contents Lists).
> 
> When sk merges seperate databases, it can easily mark them so that it
> knows where the pieces came.  One way to do this would be to append a few
> digit code to the beginning of each id which describes which database it
> came from.
> 
> >       Your document browser is going to need a mechanism to uniquely
> >       identify each document.  If there's a strong probability that the
> >       document IDs are not unique then the browsers are each going to
> >       have to reinvent their own mechanism for ensuring that the
> >       documents are unique.
> >
> >      The documents URL might be appropriate, but it's a bit unwieldy.
> 
> The document URL is a very good unique identifier, and it is already being
> passed to the help browser.
> 
> > You really need a mechanism that can (as near as possible) guarantee
> > globally unique document IDs.
> 
> I'm pretty sure that only one document lives at a given URL ;)
> 
> > > >   As you can see, not a lot of information in there so far.  "lang" and
> > > >   "encoding" could also be specified on the command line, and if not
> > > >   specified, some defaults are used.
> > > >
> > > >   I think we probably also need a modification time attribute, but I haven't
> > > >   shown that here.
> > >
> > > The modification time is stored in a seperate file which keeps track of
> > > which documents are in the database and is used for database maintenance,
> > > updating, installing, uninstalling, etc.  The contents list does not care
> > > about the modification time.
> >
> > It probably should.  Apart from the issues I outline above, this then
> > also lets you do things like "Show me the documents in order, most
> > recently modified first".
> 
> This sort of filtering should be handled by sk, which knows the timestamps
> even though they are not part of the contents list.  (The contents list is
> there to sort the documents according to a given set of instructions - the
> categories file. It is not the primary source for the metadata or
> metametadata. We have a seperate metametadata file.)
> 
> > > >  ==========================================================================
> > > >   Category: Unfiled
> > > >
> > > >     The App-Name manual
> > > >
> > > >       Include meta information from the OMF here (SK ID = MD5)
> > > >
> > > >       Formats: Local HTML (split), PDF, Remote HTML (split)
> > > >  =========================================================================
> > >
> > > This may be useful for people who don't have a help-browser aside from
> > > Netscape.  Although Netscape won't know how to display most docs.
> >
> > True.  However, we should allow the Index builder flexibilit in deciding
> > what documents are included in the Index.  This may include producing a
> > special Index that only contains documents available in HTML, plain
> > text, and PDF (for example).
> 
> What do you mean by "index"?  The definition I gave for "index" in prop#3
> is what you find in the back of a book - an alphabetical list of concepts
> and links to where they are discussed in the given work.  My impression is
> that you mean the Contents List here.  Is that right?
> 
> I agree that we will want searching/filtering features in sk that allow
> people to control its behavior as you suggest.
> 
> Dan
> 
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel

Re: [Scrollkeeper-devel] Handling relocatable packages

[Dan M. <d-...@uc...>]

On Thu, 23 Nov 2000, Nik Clayton wrote:

> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> > 
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
> 
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?

On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
gives me 506 packages which are non-relocateable, the last of which is
xbill:)

I think Red Hat would be better qualified at explaining this decision.

> > The current way things are designed is that it will have one contents list
> > for the system.  
> 
> I think that's a bad idea.
> 
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does.  I'm describing
> what I think SK should do, and what the interfaces should be.  It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
>
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed.  By specifying which OMF directories you want to include,
> > these databases could be made to differ.  We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned.  This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
> 
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.

I must have been unclear above.  We definitely want to keep all of the
original metadata in OMF files and seperate from the database.  These OMF
files should never be edited after they are initially installed (and
possibly initially fixed to have the right IDENTIFIER during the
post-install script, for relocatable packages).  This way we are
guaranteed to have an authoritative copy preserved in case the databases
are corrupted and to make upgrades simpler.

What I was trying to say above is that you can specify where the database
lives with a -p <path> flag.  If I run scrollkeeper-update -p /tmp/junk/
and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
all of the OMF files in the directory(ies) it is configured to read, and
then dumps a brand new database(ie. contents list, toc, index, ...) under
/tmp/junk/. This database won't generally be any different from the one in
its normal location (/var/lib/scrollkeeper/) unless it is running under a
different configuration.  This would happen for a user who has his own
configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
should be added to the database (typically under $HOME/.scrollkeeper/db/
perhaps).

> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization.  The problem is
> > that somebody has to specify what each database contains.  So it works for
> > users, because they can hand-tweak the contents.  It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager.  So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.  
> 
> I think it does, as long as a mechanism is provided so that sk can work 
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
> 
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF.  OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
> 
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.

We definitely want to support multiple databases, each of which has a
Contents List.

I am still uncertain if we want inheritance between contents lists though.

In many cases users will not want inheritance, because they lose any
control over what they inherit.  If instead they generate their own
database, they have complete control over all its parts, including those
parts which they would have otherwise inherited.  One way they could do
this is to over-ride one or more OMF files by keeping local
copies.  Another way is to have a local categories file which differs from
the system's. This approach gives the user complete control over their
database (including their contents lists) which would be much more
difficult and awkward to achieve by inheriting.

Now we can come up with instances where inheritance is useful.  If a user
wanted all the man pages in their help system, they might be better served
by inheriting the system man page database.  They would be limited though,
in that they could not specify which sections of the man pages they want.
This would be solved by having a database for each section of man
pages.  Then these sections could be inherited by users which would be
much faster than regenerating them for each user.

I think we really need to be very clear on how this will work and how it
will be used.  Would all databases be generated with the same categories
list?  If not, how do you handle merging them?  Presumably there is a
"default" categories list which gets enforced on the final contents list.

I think performance measurements would be useful here.  The example above
of using inherited databases with man pages has an advantage because
updating the database is faster.  It may save disk space, but this is
probably negligable. It has the downside of slowing down run-time
performance and making things substantially more complex and fragile.

> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document.  The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.

If I understood you, the "system" (ie nothing under /home) would have
multiple databases.  So when documentation is registered, it must be
determined which database to register it in.  Since this must be passed to
the registration script, it is determined by the packager based on
information about the content of the docs.  Thus, it is based on document
metadata.  (eg. this is a man page so it goes in the man page database,
this one is a KDE doc so it goes in the KDE database, ...)  This type of
metadata is stored in the OMF file already.

What various databases do you envision?

[SNIP]

> > What is the advantage of breaking the main database into pieces?
> 
> Flexibility for the future.  
> 
> > >   2.  Maintaining a TOC from the Contents List
> > > 
> > >   3.  Maintaining an Index from the Contents List
> > > 
> > >   4.  Maintaining a Categories list
> > 
> > Well, this is somewhat debatable and worthy of discussion.  Somebody has
> > to provide and maintain the categories list.  On one side, we can just
> > define it as part of OMF/sk and install a static list with sk.  Then no
> > maintenance is necessary.  Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
> > sys admin could drop in a replacement if they want.  For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
> 
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist.  I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.

We could provide an option to not use categories lists.  We definitely
should provide the ability to use them however, and we should specify a
suggested list.  

> However, that's not the same thing as saying
> 
>   If a categories file exists (in some, as yet undecided format), SK
>   should have the option of warning the SA that they're about to install
>   a document with a category that does not currently exist, and asking
>   whether they want to continue.  Naturally, the SA should have the
>   option of overriding this check with a command line flag.

We could provide a warning and even an interactive mode as an option.  

> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > >   application will have the responsibility for handling user queries of the
> > >   SK TOC and Index files ]
> > 
> > sk should export searching functionality. 
> 
> How?  libsk.so?  By having a program that accepts search parameters and
> returns XML with the results?  Some other mechanism?

We definitely want libsk.so.  I'm not sure if we would want a command-line
program for this.

> > > Activities (2) and (3) require a Contents List.  Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> > > maintain several different TOCs and Indices from several different Contents 
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> > 
> > For a given database, I agree.  Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one.  Is
> > this what you have in mind?  If you think we want more than one contents
> > list in the system's database, please explain why.
> 
> Flexibility and efficiency.  Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on.  That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.

It will be big.  Hopefully we can start running some tests on performance
some time soon to see how big of an issue this becomes.  It is not clear
that merging two medium sized lists into a big list is any better than
starting with a big list.  You still have a big list in the end, only you
save the part where you have to process and store two medium lists all the
time.

> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man.  So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.

This would speed things up a bit when one installs a new OMF file if man
pages were treated the same as other documents.  However processing of man
pages are done seperate from other applications since the OMF files for
most (all?) man pages are generated and can be stored in a special
location.  This way the man pages are updated with cron each night (which
must also make sure to create OMF metadata for each man page) and the
normal docs are registered whenever an application installs.  Because they
are processed at different times they do not slow each other down.

Aside from rebuilding the database in cases where it becomes corrupt, I'd
like this to be a zero maintenance system that works out of the box.  I
don't think we should focus on unusual things a SA may want to
do.  Instead we should make it work well and in a robust manner so the SA
won't be tempted to try to "fix" it.

> > > What sort of documentation can the Contents List going to point to?
> > > 
> > >   1.  Locally installed standard documentation that comes with the system,
> > >       such as man pages, or Info documents.
> > 
> > Perhaps.  I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below.  For man and info, we would have to update this information
> > off of cron unfortunately.  So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
> 
> I don't quite agree.  As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
> 
> First, the user tells SK about the documentation.
> 
> Second, the user assigns some OMF to the documentation.
> 
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.

Why do we need the former step?  The OMF data includes the IDENTIFIER,
which makes the first step unnecessary.

[SNIP]

> Note that I don't mean that SK goes and writes a new, master contents
> list file.  Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).

If we import a contents list it could only have documents on the Net on it
or the docs would generally be unreachable.  However this could be a very
useful feature.  You could imagine a given project publishing a database
and docs on the Net and then importing them into your local
system.  Without the ability to inherit contents list one would have to
install the metadata locally.  This would be possible and could even be
automated.  We would need to consider this carefully to determine which
solution is simpler and more robust.

> > > DD:  sk must have a run time option to allow the end user to select which
> > >      TOC to update.
> > 
> > The databases are stored together.  So, this should be specified at the
> > same time as the contents list part of the database.  I don't think I want
> > my contents list coming from one database and my toc from another.
> > 
> > (Or am I misunderstanding you?  I'm not sure what you mean by "update" a
> > TOC.)
> 
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC.  That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.

Yes.

> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.

Not if only one contents list/toc pair live in each database directory.

> Or are you saying that the ToC should be stored within the Contents List?

No.

[SNIP]

> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.

The only way scrollkeeper knows that two docs of different
versions/formats/languages are otherwise the same is using the extra OMF
element (which doesn't yet exist) which assigns them an ISBN-type code
which is the same.  Since they have different IDENTIFIERS (and other
metadata), sk knows they are different documents.

[SNIP]

> > Why do we specify the format on the command line?  The FORMAT is specified
> > in the OMF already.  The packager should just install and register the
> > metadata instead of putting it on the command line.
> 
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we 
> give it.
> 
> This separation is necessary to support documents that have no separate
> OMF.  Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.

Perhaps we agree on this after all and are just talking past each
other.  So for docs with OMF we have two steps:
1) update the IDENTIFIER to point to the right URL
2) register the OMF file with sk

Really, #1 is only necessary for local docs (as opposed to remote
ones) and for relocatable packages.

For docs without OMF, we do two steps:
1) generate the best OMF file we can given the doc
2) register the OMF file with sk

> > Here I assume (hope?;)  that you mean the database directory and not the
> > contents list.  If so, we already do this with the "-p" flag.
> 
> Right.

So we would have multiple databases, each with exactly one Contents List,
TOC directory, etc., right?

> I'd like to see a mechanism to specify it on the command line (apart
> from anything else, it's easier to script).  sk-build-omf, as you
> suggest above, seems like a good compromise.

:)

> > Right now we just number them, which guarantees a unique identifier.  
> 
> Two problems with this:
> 
>   1.  Probably won't scale efficiently as the Contents List gets larger.
> 
>   2.  Makes it impossible to merge two separate Contents Lists, from two
>       or more sites (say you're on a univesity campus, where the biology
>       department and the chemistry department both make available their
>       list of documentation in two different Scroll Keeper Contents Lists).

When sk merges seperate databases, it can easily mark them so that it
knows where the pieces came.  One way to do this would be to append a few
digit code to the beginning of each id which describes which database it
came from.

>       Your document browser is going to need a mechanism to uniquely
>       identify each document.  If there's a strong probability that the
>       document IDs are not unique then the browsers are each going to
>       have to reinvent their own mechanism for ensuring that the
>       documents are unique.
>
>      The documents URL might be appropriate, but it's a bit unwieldy.

The document URL is a very good unique identifier, and it is already being
passed to the help browser.

> You really need a mechanism that can (as near as possible) guarantee
> globally unique document IDs.

I'm pretty sure that only one document lives at a given URL ;)

> > >   As you can see, not a lot of information in there so far.  "lang" and
> > >   "encoding" could also be specified on the command line, and if not 
> > >   specified, some defaults are used.
> > > 
> > >   I think we probably also need a modification time attribute, but I haven't
> > >   shown that here.
> > 
> > The modification time is stored in a seperate file which keeps track of
> > which documents are in the database and is used for database maintenance,
> > updating, installing, uninstalling, etc.  The contents list does not care
> > about the modification time.
> 
> It probably should.  Apart from the issues I outline above, this then
> also lets you do things like "Show me the documents in order, most
> recently modified first".

This sort of filtering should be handled by sk, which knows the timestamps
even though they are not part of the contents list.  (The contents list is
there to sort the documents according to a given set of instructions - the
categories file. It is not the primary source for the metadata or
metametadata. We have a seperate metametadata file.)

> > >  ==========================================================================
> > >   Category: Unfiled
> > > 
> > >     The App-Name manual
> > > 
> > >       Include meta information from the OMF here (SK ID = MD5)
> > > 
> > >       Formats: Local HTML (split), PDF, Remote HTML (split)
> > >  =========================================================================
> > 
> > This may be useful for people who don't have a help-browser aside from
> > Netscape.  Although Netscape won't know how to display most docs.
> 
> True.  However, we should allow the Index builder flexibilit in deciding
> what documents are included in the Index.  This may include producing a
> special Index that only contains documents available in HTML, plain
> text, and PDF (for example).

What do you mean by "index"?  The definition I gave for "index" in prop#3
is what you find in the back of a book - an alphabetical list of concepts
and links to where they are discussed in the given work.  My impression is
that you mean the Contents List here.  Is that right?

I agree that we will want searching/filtering features in sk that allow
people to control its behavior as you suggest.

Dan

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > There should be no such thing as a non-relocatable package.
> 
> Both relocatable and non-relocatable packages exist, so scrollkeeper
> should support them both in the simplest possible way.

Can you give me an example of non-relocatable package, and it's
justification for being non-relocatable?

> The current way things are designed is that it will have one contents list
> for the system.  

I think that's a bad idea.

Also, I should be clear here -- I'm not talking about how the system is
currently designed, or what the code currently does.  I'm describing
what I think SK should do, and what the interfaces should be.  It's kind
of hard to evaluate the existing functionality at the moment, because I
haven't seen a roadmap that lists explicit design decisions, and things
that will and won't be supported.

> Of course you can always create another one somewhere
> else using the -p flag, but generally they would contain the same
> information and only the default one will get updated each time a document
> is installed.  By specifying which OMF directories you want to include,
> these databases could be made to differ.  We don't currently have a
> mechanism for supporting or specifying multiple OMF directories, but that
> would be fairly straightforward and is planned.  This will make it very
> simple for users to have their own local databases which vary from the
> system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> which add to or over-ride the system defaults.

This now sounds different from design doc #3, which, IIRC, suggested
that the contents list, and the OMF information, would be kept separate.

> It is not clear how and why one would want to use multiple contents lists
> except for individual users and/or hand customization.  The problem is
> that somebody has to specify what each database contains.  So it works for
> users, because they can hand-tweak the contents.  It doesn't work very
> well for system-wide databases, since everything must be specified by the
> packager.  So while you could imagine GNOME documents being kept in one
> list and KDE documents being kept in another list, this doesn't really
> achieve anything.  

I think it does, as long as a mechanism is provided so that sk can work 
out where to put the information if the user doesn't indicate otherwise.
A config file that specifies the default Contents List should suffice.

> The information which would be passed on the command
> line to sk to indicate which list the doc goes in is really metadata which
> belongs in the OMF.  OMF and XML provide everything we need to have one
> database and understand the scope of each piece. (eg. whether it is a
> GNOME doc or KDE doc.)

If you support multiple Contents Lists (and you have to, just to support
end users installing docs and not having write access to the system wide
Contents List) then you need to be able to specify which Contents List
to use.

Yes, you could put that information in the OMF, but, as I understand it,
the OMF contains meta information about the document.  The location of
the Contents List is not meta information about the document, so should
not be in the OMF.

Also, consider a regular user who occasionally installs applications and
documentation in to $HOME.

It's much easier for them to add a flag to a command line (or, better
still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
than it is for them to edit OMF files before installing them.

> What is the advantage of breaking the main database into pieces?

Flexibility for the future.  

> >   2.  Maintaining a TOC from the Contents List
> > 
> >   3.  Maintaining an Index from the Contents List
> > 
> >   4.  Maintaining a Categories list
> 
> Well, this is somewhat debatable and worthy of discussion.  Somebody has
> to provide and maintain the categories list.  On one side, we can just
> define it as part of OMF/sk and install a static list with sk.  Then no
> maintenance is necessary.  Or, we may consider having this specified by a
> configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
> sys admin could drop in a replacement if they want.  For example you could
> chop out all the HOWTO's and man pages if you know newbies who never touch
> a command line will be using your machine.

I don't think SK should mandate the existence of a categories list, or
mandate that certain categories should exist.  I expect that some well
defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
"Gnome") will spring up.

However, that's not the same thing as saying

  If a categories file exists (in some, as yet undecided format), SK
  should have the option of warning the SA that they're about to install
  a document with a category that does not currently exist, and asking
  whether they want to continue.  Naturally, the SA should have the
  option of overriding this check with a command line flag.

> > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> >   application will have the responsibility for handling user queries of the
> >   SK TOC and Index files ]
> 
> sk should export searching functionality. 

How?  libsk.so?  By having a program that accepts search parameters and
returns XML with the results?  Some other mechanism?

> > Activities (2) and (3) require a Contents List.  Note that a don't say "the
> > {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> > maintain several different TOCs and Indices from several different Contents 
> > Lists on the same host (for example, a regular user might want to run sk
> > to maintain their own index and TOC of ~/my-docs).
> 
> For a given database, I agree.  Eventually, I see the system having one
> database (with one contents list) and possibly each user having one.  Is
> this what you have in mind?  If you think we want more than one contents
> list in the system's database, please explain why.

Flexibility and efficiency.  Suppose you have one big contents list that
contains information about all the documentation that's installed on the
system -- man pages, readmes, faqs, and so on.  That's going to be a
very big contents list, particularly when apps like Gnome and KDE get in
on the act.

For example, on a BSD system, the contents of /usr/share/man change once
in a blue moon, as third party manual pages generally live under
/usr/local/share/man.  So the admin might want to create a Contents List
for /usr/share/man once, and keep it separate, because the SA knows its
never going to change, so it doesn't need to be included in any
processes SK undertakes to ensure that the Contents List is sync with
the filesystem.

> > What sort of documentation can the Contents List going to point to?
> > 
> >   1.  Locally installed standard documentation that comes with the system,
> >       such as man pages, or Info documents.
> 
> Perhaps.  I think we probably do want to include this information, using
> automagically generated (and therefore incomplete) OMF metadata as you
> suggest below.  For man and info, we would have to update this information
> off of cron unfortunately.  So there will be an unavoidable delay between
> when a document is installed and when it is registered with sk.

I don't quite agree.  As I say below, I think installing a new document
should be a two step process, at least conceptually (as it could be
carried out by one command).

First, the user tells SK about the documentation.

Second, the user assigns some OMF to the documentation.

This is what I'm getting at with the sk-install-doc and sk-install-omf
command separation.

Consider the man page, for example.  If I want to tell SK about all my
section 1 man pages, I'd like to be able to do something like

    cd /usr/share/man/man1
    sk-install-doc *.1

which would tell SK about all my section 1 manual pages.  Actually, what
I'd probably do is something like

    sk-install-doc -category "Man Pages:Section 1" *.1

to assign a category to them.  At this point, there's no OMF for them,
but SK can synthesise the bare bones from the filename and the category.

Suppose, then, that I want to add some OMF for ls.1.  I might do
something like this;

    sk-extract-omf ls.1 > ls.omf
    vi ls.omf
    [edit edit edit]
    sk-install-omf ls.1 ls.omf

which would pull out the OMF that SK synthesised for me in the earlier
step (presumably as XML).  I can then edit it, and then install the
edited OMF back in to SK's control.

I think it's highly likely that *roff -man (and in particular, *roff
-mdoc) is going to be easy to extract OMF from.  So what we might end up
with is a command to extract OMF from -man formatted documents.  It
might work like this;

    cd /usr/share/man/man1
    foreach man in (*.1)
        extract-omf -source man $man > `basename $man .1`.omf
    end

which would iterate over *.1, extracting the OMF and writing it to
*.omf.  With that done, the end user might then be able to do

    foreach man in (*.1)
        sk-install-doc -doc $man -omf `basename $man .1`.omf
    end

which could install the document, and the pregenerated OMF, into SK with
one command.

I hope the design ethos that's coming across from all of this is to have
lots of small tools, each of which does one thing.

> >   2.  Locally installed system documentation, installed by the SA team.
> 
> SA =? sys admin

Yes.

> >   3.  Locally installed documentation, installed by the user.
> 
> Yes.  Although like much of this discussion, this feature will come in a
> post-0.1 release.

As I say, I'm not thinking in terms of 'release' yet.  I'd like to get
some specific functionality nailed down and agreed on.

[ Incidentally, and talking about the implementation -- one of the
  things that occured to me last night; sk could have a command
  structure like that of cvs, where instead of having a bunch of
  sk-foo-bar commands, you have one 'sk' command, that takes options,
  and then subcommands, which also take options.

      sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
      sk install-omf /path/to/omf

  and so on. ]

> > I think it is fair to say that, when run, sk will need to construct it's
> > global Contents List by parsing multiple, smaller, contents lists.  In much
> > the same way that SGML catalog files can include one another at the moment.
> 
> Why? It sounds very nice and wonderful.  However, this adds a layer of
> complexity without any obvious benefits.

As above.

Note that I don't mean that SK goes and writes a new, master contents
list file.  Just that, in memory, SK is capable of aggregating
information from multiple Contents Lists (which should be accessible by
URL, so that remote Contents Lists are feasible).

> > DD:  sk must have a run time option to allow the end user to select which
> >      TOC to update.
> 
> The databases are stored together.  So, this should be specified at the
> same time as the contents list part of the database.  I don't think I want
> my contents list coming from one database and my toc from another.
> 
> (Or am I misunderstanding you?  I'm not sure what you mean by "update" a
> TOC.)

It's my understanding that there's a 1:1 mapping between a Contents List
and a ToC.  That is, you can't have a ToC that includes information
about documents that aren't in the Contents List.

So if you have multiple Contents Lists, you need to have multiple ToCs
(and Indexes), and a way of saying that >this< Contents List was used to
produce >This< ToC.

Or are you saying that the ToC should be stored within the Contents List?

> > DD:  sk must have a run time option to allow the end user to select which
> >      Index to update.
> 
> (as above)

Ditto :-)

> > DD:  One of the sk commands must be to generate a usable Contents List 
> >      (which really means {X}HTML version, to start with) given a Contents
> >      List which is in sk internal format.
> 
> Everything is XML right now, so this is not an issue.  If we decide
> another internal format is better we should still export XML to the
> browser.  As for HTML, this is possible and may be a nice interface for
> people who want to create a very minimalistic web browser.

All the world is not yet XML, and not everyone runs KDE or Gnome.  Being
able to convert the ToCs and Indices to other formats is vital.
Fortunately, since the ToC and Indices are XML, this should be easy,
using something like XSLT, so we can probably just consider this
functionality that we get for free.

> > DD:  It is possible for a document to change, but for the OMF information
> >      to be neglected.  sk should maintain a modification time stamp for
> >      each document and its corresponding OMF, to allow the end user to 
> >      determine when that might have happened, and to correct for it.
> 
> For the current version of sk, which does not yet allow for remote
> documents, only the OMF file's timestamp is watched.  This is because if
> somebody edits the doc and not the OMF, then it is their fault for not
> updating the OMF, not sk's.  In reality whenever you install a new version
> of a package, even if the OMF is not updated (as it should), it still
> updates the old version so it gets a new timestamp and the new doc gets
> registered despite any negligence on the part of the maintainer to update
> the OMF (or the doc for that matter;).
> 
> This mainly becomes an issue for:
> 1) remote docs
> 2) docs which users write and modify themselves.
> 
> So we may indeed need to start tracking the doc timestamps when we support
> these features.  We have to be careful about constantly checking the
> timestamps on a bunch of docs which are spread around the internet
> though.  There will be many cases where this task cannot complete, or else
> takes an unacceptable amount of time.

This should be configurable behaviour, something like;

    sk --check-remote-timestamps --remote-timeout=30

Actually, this implies another DD

DD:  sk must track the validity of the OMF in relation to it's document.
     OMF might be:

        valid      This OMF was installed with the document, and neither
	           the document or the OMF's timestamps have changed.

        invalid    The document and OMF have different timestamps

	unknown    The document and OMF have the same timestamps, but
	           the most recent attempt to confirm this (particularly
		   for remote documents) failed.

     sk should be able to report on documents who have OMF which is not
     valid, so the SA can investigate further.

> > DD:  One document, multiple output formats.  The user might have installed
> >      the same document three times, once as HTML, once as PDF, and once as
> >      plain text.  The three installations are the same document, and have
> >      the same OMF associated with them.
> 
> No. These definitely do not have the same OMF, unless they are identical
> copies.  The OMF file specifies the document format in FORMAT.DTD and
> FORMAT.MIME.  The docs also have different IDENTIFIER's.
> 
> (We may decide to expand this a little bit.  FORMAT.STYLESHEET may be
> nice, as it would allow the doc authors to dictate the appearance of their
> doc.  So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> icon, etc.)
> 
> >      sk should treat this as one document with three formats, rather than
> >      three separate documents.     
> 
> If you look in the ldp-meta archives from several months ago, I proposed
> we add a document id to the OMF.  This would allow us to identify if two
> documents are "The-Same".  I call two documents "The-Same" if they are the
> same document only different versions, languages, formats, etc.  If two
> documents have the same title, they are not necessarily
> "The-Same".  Likewise, if I change the title of my document at some point
> in time, I can have two docs which are "The-Same" but not having the same
> title.
> 
> Obviously this sort of information will be very nice to have.  You can do
> things like "Get me this document in language xxx." or "Show me the latest
> version of this document on the Net."  or "Show me the previous version of
> this document from the Net."
> 
> >      [ This is probably going to be the most contentious piece of this 
> >        writeup, and it might change.  In particular, it also suggests that
> >        translations of a document should be treated as the same document,
> >        just in a different language, and I'm not sure that's a good idea.
> >        Then again, it might be. ]
> 
> I guess we were thinking along similar lines :)

Do you agree that making the distinction between a document, and one or
more instances of that document (different formats, different languages)
is worthwhile.

> > There are a number of tasks that the sk end user is going to want to tell 
> > sk to do.  They are;
> > 
> > Install new documentation that comes bundled with OMF
> > 
> >   The user has just downloaded and installed a package that comes with one
> >   or more bundled documents, and the package maintainer has thoughtfully
> >   provided an OMF file for each document.
> > 
> >   Each document has been provided in HTML and PDF format.
> > 
> >   Suppose the user installs the HTML documentation first, and installs it
> >   it into /usr/local/share/doc/app-name/html/, where normal file is called
> >   index.html.
> > 
> >   [ Everywhere I say "the user runs" I really mean "The package maintainer
> >     does this by means of a post-install script that they've already written.
> >     It is assumed that this post-install script knows the directories that
> >     documentation has been installed in, by virtue of being part of the 
> >     package that did the installation in the first place. ]
> > 
> >   The user runs 
> > 
> >      sk-install-doc -format html-split -db /var/db/sk/master.xml \
> >          /usr/local/share/doc/app-name/html/index.html
> 
> Why do we specify the format on the command line?  The FORMAT is specified
> in the OMF already.  The packager should just install and register the
> metadata instead of putting it on the command line.

Because at this point, all we've done is install a document with no OMF.
SK has to synthesise some OMF, based on any additional information we 
give it.

This separation is necessary to support documents that have no separate
OMF.  Adding "-format" to the command line is probably going to be
easier for an end user than having to create a separate file to return
to.

> >   'sk-install-doc' registers the document.  The two (optional) arguments
> >   here are
> > 
> >      -format    The format of the installed documentation.  Might be
> >                 html-split      - Bunch of small HTML files
> >                 html            - One big HTML file
> >                 ps              - postscript
> >                 pdf             - PDF
> >                 ...             - extend as necessary (rtf, txt, pdb, ...)
> > 
> >                 If not specified, sk can try and guess what it is.
> 
> I don't think we really want this.  One could make an argument that if no
> OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
> has a bunch of arguments for each element in the OMF.  However I'm not
> sure if we want to merge all this into the same command that registers the
> document with sk.

Six of one, half a dozen of another.  Certainly, a sequence that went
something like

    sk-build-omf -format html  ... > doc.omf
    sk-install-doc <options here> index.html
    sk-install-omf <options here> doc.omf

would also work.

> >      -db        The Contents List to update.  Uses a default if not specified.
> 
> Here I assume (hope?;)  that you mean the database directory and not the
> contents list.  If so, we already do this with the "-p" flag.

Right.

> >   [ Note:  This also needs a category option, so that the end user can
> > 	   specify which categories the document goes in to.  By default, 
> > 	   it goes in to "unfiled". ]
> 
> Here we go, implementing the OMF on the command-line ;)
> We should put all this metadata in the OMF metadata file.

I'd like to see a mechanism to specify it on the command line (apart
from anything else, it's easier to script).  sk-build-omf, as you
suggest above, seems like a good compromise.

> >   This command does not physically copy the file in to place.
> > 
> >   sk-install-doc should generate a unique identifier for this document
> >   (MD5 hash of the filename?), and display this to the user.  We probably
> >   also need a standalone utility (sk-generate-id?) that, given a path, 
> >   generates an ID.
> 
> Right now we just number them, which guarantees a unique identifier.  

Two problems with this:

  1.  Probably won't scale efficiently as the Contents List gets larger.

  2.  Makes it impossible to merge two separate Contents Lists, from two
      or more sites (say you're on a univesity campus, where the biology
      department and the chemistry department both make available their
      list of documentation in two different Scroll Keeper Contents Lists).

      Your document browser is going to need a mechanism to uniquely
      identify each document.  If there's a strong probability that the
      document IDs are not unique then the browsers are each going to
      have to reinvent their own mechanism for ensuring that the
      documents are unique.

      The documents URL might be appropriate, but it's a bit unwieldy.

You really need a mechanism that can (as near as possible) guarantee
globally unique document IDs.

> >   As you can see, not a lot of information in there so far.  "lang" and
> >   "encoding" could also be specified on the command line, and if not 
> >   specified, some defaults are used.
> > 
> >   I think we probably also need a modification time attribute, but I haven't
> >   shown that here.
> 
> The modification time is stored in a seperate file which keeps track of
> which documents are in the database and is used for database maintenance,
> updating, installing, uninstalling, etc.  The contents list does not care
> about the modification time.

It probably should.  Apart from the issues I outline above, this then
also lets you do things like "Show me the documents in order, most
recently modified first".

> Hopefully we agree that sk should be as simple and robust for the packager
> and distributor as possible.  This means the least amount of typing with
> the smallest possible learning curve, with good fault tolerance.

Yes.  Having said that, we shouldn't be afraid to aggregate certain
functionality in to one tool.

For example, cp(1), mv(1), chmod(1), and chown(1) all do one task, and
do it well.  That doesn't stop install(1) from being useful in its own
right.

> >   OK, so we've told sk about the document, but we haven't pointed to
> >   any of the OMF data yet.  Recall that the OMF data is the same despite
> >   the different document formats.
> 
> Actually they are different OMF files which have different IDENTIFIER's
> and FORMAT's.

OK.  I'm going to poke around OMF in more detail, and think about this
issue some more.

> >  ==========================================================================
> >   Category: Unfiled
> > 
> >     The App-Name manual
> > 
> >       Include meta information from the OMF here (SK ID = MD5)
> > 
> >       Formats: Local HTML (split), PDF, Remote HTML (split)
> >  =========================================================================
> 
> This may be useful for people who don't have a help-browser aside from
> Netscape.  Although Netscape won't know how to display most docs.

True.  However, we should allow the Index builder flexibilit in deciding
what documents are included in the Index.  This may include producing a
special Index that only contains documents available in HTML, plain
text, and PDF (for example).

> > Generate TOC and Index
> > 
> >   sk-generate-toc -format html -db /var/db/master.xml
> > 
> >   sk-generate-index -format html -db /var/db/master.xml
> 
> As with the contents list, we could export html.  However I think most
> browsers will find XML much more useful.

As I say, this is a simple XSLT transformation.  As long as the source
format is XML this should be relatively simple.

> I'm unclear on how and why we should embed contents lists inside each
> other.  

Hopefully I've answered that.

> I'm also very scared by the idea of listing metadata on the
> command line in addition to inside the OMF metadata file.  I think this
> creates a lot more work for packagers and risks introducing a lot of
> errors and inconsistancies in the metadata without gaining anything.  The
> obvious exception is the case where an OMF file does not exist, but I
> think creating the metadata and installing it should be two seperate
> steps.

I'm only considering it for the cases where there is no OMF, and you
need to bootstrap it in to SK.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Dan M. <d-...@uc...>]

On Wed, 22 Nov 2000, Ali Abdin wrote:

> * Dan Mueth (d-...@uc...) wrote at 22:57 on 22/11/00:
> > 
> > On Wed, 22 Nov 2000, Ali Abdin wrote:
> > 
> > > On Tue, 21 Nov 2000, Bob Stayton wrote:
> > 
> > > > 1.  A predetermined complete categorization of content is
> > > > not useful or even possible.  Rather, an open framework
> > > > of general categories can be established so providers
> > > > have a place to hang their stuff.  But providers need
> > > > the ability to add subcategories.
> > > 
> > > As far as I know - anyone can put their document in any category they 
> > > like. But there should be a set of "recommended" categories for people to 
> > > use.
> > > 
> > > This happens in RPM, people can put their category as anything in their 
> > > Spec file, but usually they pick one of the "standard" ones.
> > > 
> > > Scrollkeeper will not "enforce" a strict set of categories, but I think a 
> > > recommended list of categories should be published
> > 
> > I agree with this point of view.  Scrollkeeper itself is just a software
> > package, not a specification.  The OMF is the spec for the metadata.  We
> > will develop another spec for a standard category system for the contents
> > list.  However, people should be able to use scrollkeeper with their own
> > contents list.  (eg. in a classroom or computer lab for newbies)  I'd like
> > to just point scrollkeeper to whichever categorization file it should use.
> > 
> > We need to decide what happens to docs which fall out of the
> > categorization.  We have 3 options:
> > 1) they are omitted
> > 2) they are put in a generic "other" category(ies)
> > 3) they are put in regardless of the categories file
> > 
> > We may want a configuration parameter to determine which of these 3 are
> > done, since depending on your situation you may want any of these.
> 
> I disagree. ScrollKeeper should not "point" to or know about any categories.
> It just uses whatever is specificed in the OMF file. So if the category is
> 'Audio/MP3 Player' it gets stays that, and if the category is 'ABC/XYZ' it stays
> that 'way'.
> 
> What ScrollKeeper should do is provide a "list" (on the webpage, in the docs
> section, etc.) of "recommended" categories to use.

Yes.  Nothing I said contradicts this.  I'm just saying that in addition
to publishing a "recommended" categorization system, we also accomodate
documents which fall out of this recommended categorization and
categorizations other than the recommended one.

So if a user wants to copy the standard contents list and add a section
called "DocsIWrote" or "Favorites" to it, they can do this.  Similarly,
they can (eventually) create OMF files under $HOME/.scrollkeeper/omf/
which can populate these sections they create in their contents list tree.

[SNIP]

> > It would still be nice to have these example programs, or even have a
> > complete help browser which does not depend on all the GNOME and KDE
> > libraries.  I don't personally place this high on my priorities list
> > though.  If anybody else wants to write a desktop independent browser
> > which uses ScrollKeeper then this would certainly be a Good Thing.
> 
> I have a gripe with Scrollkeeper right now...It appears that the only way to
> interface with it is to use applications/scripts. 
> 
> I thought we were going to have a library that implements this sort of
> functionality (to get the contents list, etc.) - I really don't like the idea
> of using pipe's for the help browser (a library would also be more
> efficient)...

Yes. This is the plan.  We have not worked out the details here, so if you
would like to suggest how this API would work, please do.

> Another mini-gripe...In scrollkeeper/cl/README it says 'foo.xml is the
> document metadata file' - I thought we had agreed to use 'foo.omf' instead to
> help identify the file easier?

Yes.  We should go through and make sure this is fixed everywhere.  I see
that the example package does it correctly.

> Another question...Where will be the "burden" be of searching the index? For
> example, say I want to "filter" the help files on the category 'Internet/Browsers'...Is
> this the responsibility of the help browser, or is it in Scrollkeeper's
> domain?

Most metadata searches belong in ScrollKeeper.  This is partly because
ScrollKeeper should be handling the metadata instead of the help browser,
partly to reduce the work which must be duplicated in each help browser, 
and partly because we will eventually have ScrollKeeper acting as a
"middle-man" between the help browser and metadata servers on the Net
which can handle these queries.

The particular example you give is more of filtering (as you called
it) then searching, so it isn't as obvious where this belongs.  I think it
belongs in ScrollKeeper as well.

Is this something we are likely to use any time soon?  I think at least
for Nautilus we will not need this for some time.  If anybody else wants
it, then we can move it up on the TODO list.

> The ways to implement the above are:
> 
> 1) We have an app/script that lists the entire contents, which you have to
> pipe and then search on, then present
> 2) We have an app/script that does the filtering for you, which you have to
> pipe and just present
> 3) We pass the xmlDocPtr of the entire XML database, which you have to search
> on, and then present
> 4) We pass the xmlDocPtr of the "filtered" XML database
> 
> I think its pretty obvious that I prefer #4 ;)

Sounds fine with me.  

It sounds like you have been thinking about the library ScrollKeeper
should export to help browsers.  Perhaps you could write up a proposal
describing what you think it should look like?

Dan

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Ali A. <ali...@au...>]

* Dan Mueth (d-...@uc...) wrote at 22:57 on 22/11/00:
> 
> On Wed, 22 Nov 2000, Ali Abdin wrote:
> 
> > On Tue, 21 Nov 2000, Bob Stayton wrote:
> 
> > > 1.  A predetermined complete categorization of content is
> > > not useful or even possible.  Rather, an open framework
> > > of general categories can be established so providers
> > > have a place to hang their stuff.  But providers need
> > > the ability to add subcategories.
> > 
> > As far as I know - anyone can put their document in any category they 
> > like. But there should be a set of "recommended" categories for people to 
> > use.
> > 
> > This happens in RPM, people can put their category as anything in their 
> > Spec file, but usually they pick one of the "standard" ones.
> > 
> > Scrollkeeper will not "enforce" a strict set of categories, but I think a 
> > recommended list of categories should be published
> 
> I agree with this point of view.  Scrollkeeper itself is just a software
> package, not a specification.  The OMF is the spec for the metadata.  We
> will develop another spec for a standard category system for the contents
> list.  However, people should be able to use scrollkeeper with their own
> contents list.  (eg. in a classroom or computer lab for newbies)  I'd like
> to just point scrollkeeper to whichever categorization file it should use.
> 
> We need to decide what happens to docs which fall out of the
> categorization.  We have 3 options:
> 1) they are omitted
> 2) they are put in a generic "other" category(ies)
> 3) they are put in regardless of the categories file
> 
> We may want a configuration parameter to determine which of these 3 are
> done, since depending on your situation you may want any of these.

I disagree. ScrollKeeper should not "point" to or know about any categories.
It just uses whatever is specificed in the OMF file. So if the category is
'Audio/MP3 Player' it gets stays that, and if the category is 'ABC/XYZ' it stays
that 'way'.

What ScrollKeeper should do is provide a "list" (on the webpage, in the docs
section, etc.) of "recommended" categories to use.
 
> > > 7.  Clean removal of content is very important, and harder
> > > than you think.  The removal scripts need to clean up
> > > databases, views, and search indexes.  We didn't pay enough
> > > attention to this item, and most of our bug reports came
> > > after some content was incompletely removed.
> > 
> > Scrollkeeper would only be responsible for cleaning up its own databases 
> > and various files.
> 
> The current design is very clean in this regard.  All of the sk databases
> are under $localstatedir, but the OMF files are elsewhere (I think
> $prefix/share/omf).  So to kill the database you just erase everything
> under $localstatedir/scrollkeeper.  What is really nice is that to
> "rebuild" the database from scratch, you just completely blow away
> $localstatedir/lib/scrollkeeper and then run scrollkeeper-update which
> just looks in $prefix/share/omf to find all the OMF files, uses these to
> find all the docs, and then completely generates a fresh database
> (typically in $localstatedir/lib/scrollkeeper/) :)
> 
> This is one of the reasons we want to seperate the post-installation
> scripts into two stages: One which makes sure the OMF files have the right
> IDENTIFIER, and a second which actually reads the OMF file and doc and
> enters the information into the database.  This is also why we don't want
> the latter step to take any metadata as an input.  All the metadata must
> be stored in the first step so it is included in the OMF files which will
> be the primary source for this information, not the database itself.  
> This way if we upgrade versions of sk and the database format is changed
> somewhat, or when the database is corrupted, it is trivial to generate a
> fresh and complete database.  (It also means we don't have to write
> complicated scripts which try to migrate databases from one format to
> another as sk versions increment.)
> 
> > > I do think the ScrollKeeper project should develop one
> > > or more basic presentation applications, to the degree that it
> > > can.  This will make it useful "out of the box" and
> > > encourage content providers to add OMF data to their packages.
> > > It will also test the data and the API, and lead to 
> > > incremental refinement of both.
> > 
> > I disagree. This is not ScrollKeeper's responsibility. Hopefully, the KDE 
> > or GNOME team can provide an "example" program to show off ScrollKeeper.
> > 
> > In fact, I believe Sun is working on a 'Help SideBar' for Nautilus on 
> > this (umm, they just sent me a 500kb patch for the Help SideBar (which I 
> > will examine when I get home)).
> 
> ScrollKeeper's scope is not generally to provide applications like this,
> but I also originally thought that a very basic application or two
> illustrating how one uses the ScrollKeeper API and what ScrollKeeper can
> do would be useful.  It looks like the GNOME/Nautilus folks will (or
> already have) already have a help browser using scrollkeeper, so people
> will soon be able to look at that to see how it works.

I got a patch today from a Sun dude (Mark <something>, sorry I forgot the name)
that implements ScrollKeeper for the help sidebar (while still retaining the
old functionality)...

I'll probably integrate it sometime soon.

> It would still be nice to have these example programs, or even have a
> complete help browser which does not depend on all the GNOME and KDE
> libraries.  I don't personally place this high on my priorities list
> though.  If anybody else wants to write a desktop independent browser
> which uses ScrollKeeper then this would certainly be a Good Thing.

I have a gripe with Scrollkeeper right now...It appears that the only way to
interface with it is to use applications/scripts. 

I thought we were going to have a library that implements this sort of
functionality (to get the contents list, etc.) - I really don't like the idea
of using pipe's for the help browser (a library would also be more
efficient)...

Another mini-gripe...In scrollkeeper/cl/README it says 'foo.xml is the
document metadata file' - I thought we had agreed to use 'foo.omf' instead to
help identify the file easier?

Another question...Where will be the "burden" be of searching the index? For
example, say I want to "filter" the help files on the category 'Internet/Browsers'...Is
this the responsibility of the help browser, or is it in Scrollkeeper's
domain?

The ways to implement the above are:

1) We have an app/script that lists the entire contents, which you have to
pipe and then search on, then present
2) We have an app/script that does the filtering for you, which you have to
pipe and just present
3) We pass the xmlDocPtr of the entire XML database, which you have to search
on, and then present
4) We pass the xmlDocPtr of the "filtered" XML database

I think its pretty obvious that I prefer #4 ;)

Regards,
Ali

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[David M. <dme...@lu...>]

On Wed, Nov 22, 2000 at 10:32:49AM -0600, Dan Mueth wrote:
> 
> On Wed, 22 Nov 2000, Ali Abdin wrote:
> 
> > On Tue, 21 Nov 2000, Bob Stayton wrote:
> 
> > > I do think the ScrollKeeper project should develop one
> > > or more basic presentation applications, to the degree that it
> > > can.  This will make it useful "out of the box" and
> > > encourage content providers to add OMF data to their packages.
> > > It will also test the data and the API, and lead to 
> > > incremental refinement of both.
> > 
> > I disagree. This is not ScrollKeeper's responsibility. Hopefully, the KDE 
> > or GNOME team can provide an "example" program to show off ScrollKeeper.
> > 
> > In fact, I believe Sun is working on a 'Help SideBar' for Nautilus on 
> > this (umm, they just sent me a 500kb patch for the Help SideBar (which I 
> > will examine when I get home)).
> 
> ScrollKeeper's scope is not generally to provide applications like this,
> but I also originally thought that a very basic application or two
> illustrating how one uses the ScrollKeeper API and what ScrollKeeper can
> do would be useful.  It looks like the GNOME/Nautilus folks will (or
> already have) already have a help browser using scrollkeeper, so people
> will soon be able to look at that to see how it works.
> 
> It would still be nice to have these example programs, or even have a
> complete help browser which does not depend on all the GNOME and KDE
> libraries.  I don't personally place this high on my priorities list
> though.  If anybody else wants to write a desktop independent browser
> which uses ScrollKeeper then this would certainly be a Good Thing.

It would be nice to put together a web front end as the
"example" program. Certainly, somebody should provide a sk interface on
the web, and it might as well be sk.

Doing a real production front-end is a dogfood issue, although there seems
to be lots of crossover here with the other doc projects, so maybe that
isn't so important.

Regards,


-- 
Dr. David C. Merrill
Linux Documentation Project
Collection Editor & Coordinator

Homepage: www.lupercalia.net
LDP:      www.linuxdoc.org

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Dan M. <d-...@uc...>]

On Wed, 22 Nov 2000, Ali Abdin wrote:

> On Tue, 21 Nov 2000, Bob Stayton wrote:

> > 1.  A predetermined complete categorization of content is
> > not useful or even possible.  Rather, an open framework
> > of general categories can be established so providers
> > have a place to hang their stuff.  But providers need
> > the ability to add subcategories.
> 
> As far as I know - anyone can put their document in any category they 
> like. But there should be a set of "recommended" categories for people to 
> use.
> 
> This happens in RPM, people can put their category as anything in their 
> Spec file, but usually they pick one of the "standard" ones.
> 
> Scrollkeeper will not "enforce" a strict set of categories, but I think a 
> recommended list of categories should be published

I agree with this point of view.  Scrollkeeper itself is just a software
package, not a specification.  The OMF is the spec for the metadata.  We
will develop another spec for a standard category system for the contents
list.  However, people should be able to use scrollkeeper with their own
contents list.  (eg. in a classroom or computer lab for newbies)  I'd like
to just point scrollkeeper to whichever categorization file it should use.

We need to decide what happens to docs which fall out of the
categorization.  We have 3 options:
1) they are omitted
2) they are put in a generic "other" category(ies)
3) they are put in regardless of the categories file

We may want a configuration parameter to determine which of these 3 are
done, since depending on your situation you may want any of these.

> > 7.  Clean removal of content is very important, and harder
> > than you think.  The removal scripts need to clean up
> > databases, views, and search indexes.  We didn't pay enough
> > attention to this item, and most of our bug reports came
> > after some content was incompletely removed.
> 
> Scrollkeeper would only be responsible for cleaning up its own databases 
> and various files.

The current design is very clean in this regard.  All of the sk databases
are under $localstatedir, but the OMF files are elsewhere (I think
$prefix/share/omf).  So to kill the database you just erase everything
under $localstatedir/scrollkeeper.  What is really nice is that to
"rebuild" the database from scratch, you just completely blow away
$localstatedir/lib/scrollkeeper and then run scrollkeeper-update which
just looks in $prefix/share/omf to find all the OMF files, uses these to
find all the docs, and then completely generates a fresh database
(typically in $localstatedir/lib/scrollkeeper/) :)

This is one of the reasons we want to seperate the post-installation
scripts into two stages: One which makes sure the OMF files have the right
IDENTIFIER, and a second which actually reads the OMF file and doc and
enters the information into the database.  This is also why we don't want
the latter step to take any metadata as an input.  All the metadata must
be stored in the first step so it is included in the OMF files which will
be the primary source for this information, not the database itself.  
This way if we upgrade versions of sk and the database format is changed
somewhat, or when the database is corrupted, it is trivial to generate a
fresh and complete database.  (It also means we don't have to write
complicated scripts which try to migrate databases from one format to
another as sk versions increment.)

> > I do think the ScrollKeeper project should develop one
> > or more basic presentation applications, to the degree that it
> > can.  This will make it useful "out of the box" and
> > encourage content providers to add OMF data to their packages.
> > It will also test the data and the API, and lead to 
> > incremental refinement of both.
> 
> I disagree. This is not ScrollKeeper's responsibility. Hopefully, the KDE 
> or GNOME team can provide an "example" program to show off ScrollKeeper.
> 
> In fact, I believe Sun is working on a 'Help SideBar' for Nautilus on 
> this (umm, they just sent me a 500kb patch for the Help SideBar (which I 
> will examine when I get home)).

ScrollKeeper's scope is not generally to provide applications like this,
but I also originally thought that a very basic application or two
illustrating how one uses the ScrollKeeper API and what ScrollKeeper can
do would be useful.  It looks like the GNOME/Nautilus folks will (or
already have) already have a help browser using scrollkeeper, so people
will soon be able to look at that to see how it works.

It would still be nice to have these example programs, or even have a
complete help browser which does not depend on all the GNOME and KDE
libraries.  I don't personally place this high on my priorities list
though.  If anybody else wants to write a desktop independent browser
which uses ScrollKeeper then this would certainly be a Good Thing.

Dan

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Laszlo K. <las...@su...>]

Thanks Bob, this is extremely useful. In addition to what Ali already
replied:

[snip]

> 2.  There's more than one way to organize and
> present the content.  Multiple different views can
> be layered on top of the content collection.  This requires
> looking at the content from the point of view of the multiple
> different users, not just the content provider.
If you think about multiple views of the database then we are thinking
about it. On Solaris this is named the network, system, user layering
(probably the same on Linux). 

> 7.  Clean removal of content is very important, and harder
> than you think.  The removal scripts need to clean up
> databases, views, and search indexes.  We didn't pay enough
> attention to this item, and most of our bug reports came
> after some content was incompletely removed.
At package uninstall this works for removal from the database and
removal of TOCs. Scrollkeeper will trace these things rather than
postuninstall scripts written by the packager. This seems to be better.
In addition when Scrollkeeper is uninstalled it will delete everything
related to it.

Laszlo

Re: [Scrollkeeper-devel] system similar to ScrollKeeper

[Ali A. <ALI...@au...>]

On Tue, 21 Nov 2000, Bob Stayton wrote:

> This mail describes the ODSSI (Online Documentation System
> Services Interface) system that I mentioned in an earlier
> mail. I'm sending this because ODSSI has goals similar to
> ScrollKeeper.  Let me be clear that this is *not* a
> proposal for ScrollKeeper, nor I'm not expecting any
> reviews of our system (believe me, with three years
> of experience using it, I'm well aware of its
> weaknesses 8^). This mail is simply a description of one
> actual implementation of a similar system.  I'll present
> what we did, and then describe how I think it relates to
> ScrollKeeper.  You can skip down to "How this relates
> to ScrollKeeper" if you don't want all the details.

That was a /long/ mail :)
[snip]

> But I think there were some lessons learned that do apply to
> ScrollKeeper:
> 
> 1.  A predetermined complete categorization of content is
> not useful or even possible.  Rather, an open framework
> of general categories can be established so providers
> have a place to hang their stuff.  But providers need
> the ability to add subcategories.

As far as I know - anyone can put their document in any category they 
like. But there should be a set of "recommended" categories for people to 
use.

This happens in RPM, people can put their category as anything in their 
Spec file, but usually they pick one of the "standard" ones.

Scrollkeeper will not "enforce" a strict set of categories, but I think a 
recommended list of categories should be published
 
> 2.  There's more than one way to organize and
> present the content.  Multiple different views can
> be layered on top of the content collection.  This requires
> looking at the content from the point of view of the multiple
> different users, not just the content provider.

Scrollkeeper is not responsible for 'displaying' the help. It is only for 
keeping track of the docs on your system (and allowing metadata for docs).
 
> 3.  Minimize the requirements on providers.  They should
> just need to provide standard metadata and then call a single
> post-install script already available on the system.
> That script would take various actions based on the
> metadata.  In a corollary, minimize the demands on
> system administrators, since they generally have little
> time to manage documentation.

I agree - make it as simple as possible for packagers, doc authors, 
program authors, etc.

> 4.  The system needs to handle content that has incomplete
> metadata.  Its degree of participation and integration into
> the system may be reduced, but it should not be excluded.
> Our system went even further and processed content with
> no metadata and no calls to the postinstall scripts, but
> that's because we could locate all the content under
> DocumentRoot.

I think this is okay - you just keep "empty" XML tags of the non-exiting 
metadata information

> 5.  Metadata can be versioned, so new metadata can supercede
> old without installing new versions of the doc.  You can
> even mark some doc as obsolete without actually removing it.

I believe this is possible with the OMF. 
 
> 6.  The system needs to take into account native language
> translations, and provide a system of fallback when some
> content is not available in a given language.  

This is also taken care of I believe

> 7.  Clean removal of content is very important, and harder
> than you think.  The removal scripts need to clean up
> databases, views, and search indexes.  We didn't pay enough
> attention to this item, and most of our bug reports came
> after some content was incompletely removed.

Scrollkeeper would only be responsible for cleaning up its own databases 
and various files.

> I think many of these items have already been proposed
> and discussed in the design so far.  
> 
> Conclusions
> -----------------
> Based on my experience to date, and at the risk of stepping
> on someone's toes, I suggest that ScrollKeeper
> divide its efforts into two areas:
> 
> 1.  Inventorying doc content on a system.
> 
> 2.  Presenting that content in various ways.
> 
> The inventorying effort includes defining the OMF metadata,
> creating a database to house the data, and defining an API
> for accessing the data.  This effort could also include
> methods for finding non-registered doc (for example,
> scanning RPMs for documentation sections), and fallback
> processes for discovering and filling in missing metadata
> for various file types.  The goal here is to define standard
> methods for recording and reading the metadata.
> 
> The presentation effort can go in many different directions
> to meet many different needs.  I don't think the
> ScrollKeeper project can anticipate all the uses for a
> documentation database like this, nor can it develop all
> applications that use it.  One site will want to use a
> Verity search engine to integrate doc with their other
> searchable content, while another site will want to create
> a website of documentation and index it with htdig, and a
> third will load all the information into their existing
> object-oriented database.  If there is a standard API for
> extracting information from the ScrollKeeper database, then
> those applications become much easier to do.

ScrollKeeper is in no way responsibly for the presentation of the doc. It 
can and should only provide the URI of the document to the "Help 
Browser" and based on this the browser presents the doc in any way it 
wishes. 

Also - for displaying the Index of Documents, is the browser's 
responsibility. ScrollKeeper can just pass the XML file to the help 
browser so it does its own parsing of the document (i.e. then the browser 
can add various functionality such as List only the documents in 
'Music/Players' or 'List only SGML docs')

Optionally, ScrollKeeper could pass xmlDocPtr's to the applications that 
use it. This alows /ScrollKeeper/ to do the searching and construction of 
the XML tree internally. The Help Browser folks would only need to 
"parse" the returned xmlDocPtr.

> I do think the ScrollKeeper project should develop one
> or more basic presentation applications, to the degree that it
> can.  This will make it useful "out of the box" and
> encourage content providers to add OMF data to their packages.
> It will also test the data and the API, and lead to 
> incremental refinement of both.

I disagree. This is not ScrollKeeper's responsibility. Hopefully, the KDE 
or GNOME team can provide an "example" program to show off ScrollKeeper.

In fact, I believe Sun is working on a 'Help SideBar' for Nautilus on 
this (umm, they just sent me a 500kb patch for the Help SideBar (which I 
will examine when I get home)).

> I think the ScrollKeeper project is already doing most
> of this. I would just like to see a clear distinction
> between forming the database and developing the applications
> from it.  It would make it easier to divide up the work,
> and clarify what needs to be delivered.

ScrollKeeper shouldn't have anything to do with developing the 
application (for example, which widget-set should it use? Gtk+/Qt? How 
would it display the HTML (write our own HTML parser???) use GtkHtml? (uh 
oh, KDE won't be happy?) use Mozilla? (ScrollKeeper requires 
mozilla?!?!)) - Not to mention you have to worry about cross-platform 
stuff, etc.

ScrollKeeper is only a project for indexing the docs on your system. 
It is the responsibility of the various "projects" (KDE, GNOME, etc.) to 
provide Help browsers that integrate with it nicely. Some other group, 
could create a command-line interface to it if they wish (actually - it 
might be nice to have a command-line interface to scrollkeeper...)


Regards,
Ali

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

> The current way things are designed is that it will have one contents list
> for the system.  Of course you can always create another one somewhere
> else using the -p flag, but generally they would contain the same
> information and only the default one will get updated each time a document
> is installed.  By specifying which OMF directories you want to include,
> these databases could be made to differ.  We don't currently have a
> mechanism for supporting or specifying multiple OMF directories, but that
> would be fairly straightforward and is planned.  This will make it very
> simple for users to have their own local databases which vary from the
> system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> which add to or over-ride the system defaults.
We will have to think this through. The current implementation does not
comply with the layering of information and configuration on the system
(network-wide, system-wide, user). I'll look into this in more detail
later and see what I can come up with. I don't know how well Linux
complies with this, Solaris does it very well in general. I'll check how
they do it with documentation. This is an architectural issue, it is
very important to sort it out before we go too deep in development.

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Dan M. <d-...@uc...>]

On Mon, 20 Nov 2000, Nik Clayton wrote:

> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
> 
> There should be no such thing as a non-relocatable package.

Both relocatable and non-relocatable packages exist, so scrollkeeper
should support them both in the simplest possible way.

> By all means, allow default paths to be included, but they must always 
> be overridable at installation time by the admin.
> 
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
> 
> Based on the design docs on the web, what does sk have responsibility for:
> 
> [ This is my current understanding.  Please correct any misconceptions I
>   might have.  Things I flag with 'DD' are design decisions that I think
>   fall out of the model. ]
> 
>   1.  Maintaining one or more lists of documentation that the user wants 
>       a TOC for (the "Contents List" from the most recent proposal)

The current way things are designed is that it will have one contents list
for the system.  Of course you can always create another one somewhere
else using the -p flag, but generally they would contain the same
information and only the default one will get updated each time a document
is installed.  By specifying which OMF directories you want to include,
these databases could be made to differ.  We don't currently have a
mechanism for supporting or specifying multiple OMF directories, but that
would be fairly straightforward and is planned.  This will make it very
simple for users to have their own local databases which vary from the
system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
which add to or over-ride the system defaults.

It is not clear how and why one would want to use multiple contents lists
except for individual users and/or hand customization.  The problem is
that somebody has to specify what each database contains.  So it works for
users, because they can hand-tweak the contents.  It doesn't work very
well for system-wide databases, since everything must be specified by the
packager.  So while you could imagine GNOME documents being kept in one
list and KDE documents being kept in another list, this doesn't really
achieve anything.  The information which would be passed on the command
line to sk to indicate which list the doc goes in is really metadata which
belongs in the OMF.  OMF and XML provide everything we need to have one
database and understand the scope of each piece. (eg. whether it is a
GNOME doc or KDE doc.)

Even in my example of users having their own configuration, we are never
merging contents list.  In this case the user's personal database is built
directly from OMF files just like the system-wide database.  Doing it
otherwise would make it almost impossible for a user to over-ride the
categorization of the system-wide database, over-ride or remove particular
entries in the system-wide database, or merge a doc into a part of the
database which is typically in the system-wide database.  Plus it is just
harder to do and messy.

Am I overlooking some particularly good reason to try to merge contents
list?

What is the advantage of breaking the main database into pieces?

>   2.  Maintaining a TOC from the Contents List
> 
>   3.  Maintaining an Index from the Contents List
> 
>   4.  Maintaining a Categories list

Well, this is somewhat debatable and worthy of discussion.  Somebody has
to provide and maintain the categories list.  On one side, we can just
define it as part of OMF/sk and install a static list with sk.  Then no
maintenance is necessary.  Or, we may consider having this specified by a
configuration variable (say in /etc/sk.conf or $HOME/.skrc).  This way a
sys admin could drop in a replacement if they want.  For example you could
chop out all the HOWTO's and man pages if you know newbies who never touch
a command line will be using your machine.

> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
>   application will have the responsibility for handling user queries of the
>   SK TOC and Index files ]

sk should export searching functionality. Basic metadata searches can be
implemented from scratch in sk.  We may want to rely on an external engine
for more complex searches potentially, but the help browser shouldn't have
to know or care about that.  It should interface directly with sk.

> Activities (2) and (3) require a Contents List.  Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> maintain several different TOCs and Indices from several different Contents 
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).

For a given database, I agree.  Eventually, I see the system having one
database (with one contents list) and possibly each user having one.  Is
this what you have in mind?  If you think we want more than one contents
list in the system's database, please explain why.

> What sort of documentation can the Contents List going to point to?
> 
>   1.  Locally installed standard documentation that comes with the system,
>       such as man pages, or Info documents.

Perhaps.  I think we probably do want to include this information, using
automagically generated (and therefore incomplete) OMF metadata as you
suggest below.  For man and info, we would have to update this information
off of cron unfortunately.  So there will be an unavoidable delay between
when a document is installed and when it is registered with sk.

>       QUESTION:  Is sk going to deal with non-SGML/XML documents, such as
>       these?

Definitely.  It should support arbitrary doc formats.  For some document
formats (esp. DocBook/XML), it can extract some nice info like TOC and
Index.  In fact, it will eventually be able to extract the whole OMF from
DocBook/XML docs, as Norman Walsh agreed to include any missing tags.

>   2.  Locally installed system documentation, installed by the SA team.

SA =? sys admin

>   3.  Locally installed documentation, installed by the user.

Yes.  Although like much of this discussion, this feature will come in a
post-0.1 release.

>   4.  Remote documentation on a web site.

Yes.

> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists.  In much
> the same way that SGML catalog files can include one another at the moment.

Why? It sounds very nice and wonderful.  However, this adds a layer of
complexity without any obvious benefits.

> DD:  Contents Lists must be able to refer to other Contents Lists, and sk
>      must have an option to chase the link to read the referred to Contents
>      List.

(as above)

> DD:  sk must have a run time option to allow the end user to select which
>      Contents Lists the user wants sk to process.

Yes.  If we have seperate databases under $localstatedir and $HOME, the
user should be able to (through the help browser configuration
dialog) configure which database to use.

> DD:  sk must have a run time option to allow the end user to select which
>      TOC to update.

The databases are stored together.  So, this should be specified at the
same time as the contents list part of the database.  I don't think I want
my contents list coming from one database and my toc from another.

(Or am I misunderstanding you?  I'm not sure what you mean by "update" a
TOC.)

> DD:  sk must have a run time option to allow the end user to select which
>      Index to update.

(as above)

> DD:  One of the sk commands must be to generate a usable Contents List 
>      (which really means {X}HTML version, to start with) given a Contents
>      List which is in sk internal format.

Everything is XML right now, so this is not an issue.  If we decide
another internal format is better we should still export XML to the
browser.  As for HTML, this is possible and may be a nice interface for
people who want to create a very minimalistic web browser.

> Each piece of documentation (which might really consist of tens or even 
> hundreds of files) can be considered to consist of two parts.
> 
>   1.  The documentation itself (*.html, *.png, foo.pdf, whatever).
>
>   2.  The OMF, containing the 17 (?) key pieces of meta information that
>       sk likes to keep about the documentation.  Not all 17 pieces of
>       information may necessarily be present.

Yes.  We will have to look closely at how sk handles various pieces of
data missing.

> Not all documentation the user wants to point to is going to have the OMF
> information associated with it.  This is unavoidable.
> 
> DD:  sk must work tolerably well with minimum OMF information.  At the
>      worst case, sk must do something useful when the only OMF information
>      available is the document title, and a URL pointing to the document's
>      'index' page.
> 
>      Actually, I take that back.  The only piece of information sk needs
>      is the URL.  In the worst case (with 0 OMF information) the URL 
>      becomes the title, until the end user provides some OMF, or points
>      sk at pre-existing OMF.

Good idea.

> DD:  sk should probably include some support applications which can parse
>      common documentation formats (*roff man and mdoc, GNU info, HTML,
>      LinuxDoc, LaTeX, DocBook) and extract as much OMF information from 
>      them as possible.
> 
>      In some cases this is necessary to produce bare bones OMF files that
>      will not be processed further.  In other cases, this will be to produce
>      OMF files that will be further customised and improved by the end user.

Yes :)

> DD:  sk should include some support applications that can generate TOC and
>      Index information from a variety of source formats.

yes.

> DD:  sk should not assume that the OMF information is in the same directory
>      as the document, nor should it assume that the location of the OMF
>      information can be inferred based on the document's location.  The
>      location of the OMF information must always be explicitly listed in 
>      the Contents List.
> 
>      The OMF information may not even be on the same host as the target
>      document.

Yes.  (This is in the existing design.)

> DD:  It is possible for a document to change, but for the OMF information
>      to be neglected.  sk should maintain a modification time stamp for
>      each document and its corresponding OMF, to allow the end user to 
>      determine when that might have happened, and to correct for it.

For the current version of sk, which does not yet allow for remote
documents, only the OMF file's timestamp is watched.  This is because if
somebody edits the doc and not the OMF, then it is their fault for not
updating the OMF, not sk's.  In reality whenever you install a new version
of a package, even if the OMF is not updated (as it should), it still
updates the old version so it gets a new timestamp and the new doc gets
registered despite any negligence on the part of the maintainer to update
the OMF (or the doc for that matter;).

This mainly becomes an issue for:
1) remote docs
2) docs which users write and modify themselves.

So we may indeed need to start tracking the doc timestamps when we support
these features.  We have to be careful about constantly checking the
timestamps on a bunch of docs which are spread around the internet
though.  There will be many cases where this task cannot complete, or else
takes an unacceptable amount of time.

> DD:  An application may have multiple pieces of documentation associated 
>      with it.  Applications are also not the only things that have 
>      documentation, the system does (think: man page sections).  Simply
>      storing information about a document isn't enough, you need to be
>      able to group documents into chunks (and probably group chunks into
>      chunks, or allow the same document to appear in multiple categories).  
>      I'm not going to discuss this in too much detail in this document, this
>      is the reason for "4. Maintaining a Categories list" earlier.

That is why we have a contents list and categories file.  This is all
specified by OMF metadata.

> DD:  One document, multiple output formats.  The user might have installed
>      the same document three times, once as HTML, once as PDF, and once as
>      plain text.  The three installations are the same document, and have
>      the same OMF associated with them.

No. These definitely do not have the same OMF, unless they are identical
copies.  The OMF file specifies the document format in FORMAT.DTD and
FORMAT.MIME.  The docs also have different IDENTIFIER's.

(We may decide to expand this a little bit.  FORMAT.STYLESHEET may be
nice, as it would allow the doc authors to dictate the appearance of their
doc.  So GNOME docs have a little GNOME icon, KDE docs have a little KDE
icon, etc.)

>      sk should treat this as one document with three formats, rather than
>      three separate documents.     

If you look in the ldp-meta archives from several months ago, I proposed
we add a document id to the OMF.  This would allow us to identify if two
documents are "The-Same".  I call two documents "The-Same" if they are the
same document only different versions, languages, formats, etc.  If two
documents have the same title, they are not necessarily
"The-Same".  Likewise, if I change the title of my document at some point
in time, I can have two docs which are "The-Same" but not having the same
title.

Obviously this sort of information will be very nice to have.  You can do
things like "Get me this document in language xxx." or "Show me the latest
version of this document on the Net."  or "Show me the previous version of
this document from the Net."

>      [ This is probably going to be the most contentious piece of this 
>        writeup, and it might change.  In particular, it also suggests that
>        translations of a document should be treated as the same document,
>        just in a different language, and I'm not sure that's a good idea.
>        Then again, it might be. ]

I guess we were thinking along similar lines :)

> SK Tasks
> --------
> 
> There are a number of tasks that the sk end user is going to want to tell 
> sk to do.  They are;
> 
> Install new documentation that comes bundled with OMF
> 
>   The user has just downloaded and installed a package that comes with one
>   or more bundled documents, and the package maintainer has thoughtfully
>   provided an OMF file for each document.
> 
>   Each document has been provided in HTML and PDF format.
> 
>   Suppose the user installs the HTML documentation first, and installs it
>   it into /usr/local/share/doc/app-name/html/, where normal file is called
>   index.html.
> 
>   [ Everywhere I say "the user runs" I really mean "The package maintainer
>     does this by means of a post-install script that they've already written.
>     It is assumed that this post-install script knows the directories that
>     documentation has been installed in, by virtue of being part of the 
>     package that did the installation in the first place. ]
> 
>   The user runs 
> 
>      sk-install-doc -format html-split -db /var/db/sk/master.xml \
>          /usr/local/share/doc/app-name/html/index.html

Why do we specify the format on the command line?  The FORMAT is specified
in the OMF already.  The packager should just install and register the
metadata instead of putting it on the command line.

>   'sk-install-doc' registers the document.  The two (optional) arguments
>   here are
> 
>      -format    The format of the installed documentation.  Might be
>                 html-split      - Bunch of small HTML files
>                 html            - One big HTML file
>                 ps              - postscript
>                 pdf             - PDF
>                 ...             - extend as necessary (rtf, txt, pdb, ...)
> 
>                 If not specified, sk can try and guess what it is.

I don't think we really want this.  One could make an argument that if no
OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
has a bunch of arguments for each element in the OMF.  However I'm not
sure if we want to merge all this into the same command that registers the
document with sk.

>      -db        The Contents List to update.  Uses a default if not specified.

Here I assume (hope?;)  that you mean the database directory and not the
contents list.  If so, we already do this with the "-p" flag.

>   [ Note:  This also needs a category option, so that the end user can
> 	   specify which categories the document goes in to.  By default, 
> 	   it goes in to "unfiled". ]

Here we go, implementing the OMF on the command-line ;)
We should put all this metadata in the OMF metadata file.

>   This command does not physically copy the file in to place.
> 
>   sk-install-doc should generate a unique identifier for this document
>   (MD5 hash of the filename?), and display this to the user.  We probably
>   also need a standalone utility (sk-generate-id?) that, given a path, 
>   generates an ID.

Right now we just number them, which guarantees a unique identifier.  We
used to have functions similar to the latter one, but decided to removed
them.  This was because we should not encourage the help browser to
actually use these id's since they may be modified when applications are
uninstalled or installed which could break a running help browser which
doesn't realize these changes have occured.

>   At this point, /var/db/sk/master.xml looks something like this:
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>

I think this file should hold information taken from the OMF file, as
described in the previous proposals. Duplicating OMF information with
command-line information is redundant, puts more burden on the packager,
and increases the chance of mistakes and conflicts occuring.

>   As you can see, not a lot of information in there so far.  "lang" and
>   "encoding" could also be specified on the command line, and if not 
>   specified, some defaults are used.
> 
>   I think we probably also need a modification time attribute, but I haven't
>   shown that here.

The modification time is stored in a seperate file which keeps track of
which documents are in the database and is used for database maintenance,
updating, installing, uninstalling, etc.  The contents list does not care
about the modification time.

>   Now, the user uses sk-install-doc again, but this time they install the
>   PDF file. 
> 
>      sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
>          /usr/local/share/doc/app-name/pdf/doc.pdf
> 
>   A very similar command line.  However, because this is another instance of
>   the same document, this time the user specifies the ID of the document.
>   master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Finally, the chap that wrote the application also maintains information
>   about the document on the web.  So the user decides to do the following;
> 
>     sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
>       http://www.example.com/app-name/index.html
> 
>   and master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="http://www.example.com/app-name/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Now, typically, the end user wouldn't run any of this themselves.  It
>   is the responsibility of whoever produced the package/rpm/whatever to
>   make sure that this happens.
> 
>   I understand that some people have said that that's too much work to 
>   expect package maintainers to do.  I don't think that's the case.  It 
>   doesn't matter if the original program's author, or the package maintainer
>   doesn't want to do it.  All it takes is someone to find this functionality
>   useful enough to spend the five minutes constructing the sk-* command
>   lines to do this, and submit it back to the maintainer.  These sorts of
>   commands are not going to change a great deal between package releases.

Hopefully we agree that sk should be as simple and robust for the packager
and distributor as possible.  This means the least amount of typing with
the smallest possible learning curve, with good fault tolerance.

>   [ In fact, I see this as being a differentiator between operating systems.
>     SuSE (to pick an example) might not bother generating OMF files for
>     applications that don't already provide them.  But it might be a point
>     of pride for the Debian people that they don't bundle a third party
>     application until the person making the .deb file has generated an OMF
>     file for it ]

I'd rather Suse and Debian distinguish themselves in other ways than
seeing which one can dedicate more man-hours to making a complicated
document registration system work properly ;)  Perhaps they can spend this
time improving sk or some other nifty application.

And unfortunately, it isn't a matter of who does a better job but a matter
of whether sk gets adopted or not.  If it is difficult to use, it will not
succeed.  If it is adopted, the simpler it is the more packages that will
support it and the more useful it will be to the user.

>   OK, so we've told sk about the document, but we haven't pointed to
>   any of the OMF data yet.  Recall that the OMF data is the same despite
>   the different document formats.

Actually they are different OMF files which have different IDENTIFIER's
and FORMAT's.

>   The end user (or their package management application) runs
> 
>      sk-install-omf -id MD5 app-name.omf
> 
>   Notice how the MD5 checksum (or whatever) is included on the command line,
>   so that sk can associate this OMF file with the document that was installed
>   earlier.

Having the package determine the id would likely cause problems, but can
safely be omitted since sk can manage the id's itself.

>   Now, because we've already installed the document, and sk knows where the
>   document has been installed in to, the OMF shouldn't need to contain any
>   path data.  So there's nothing for sk to need to rewrite.  sk can either
>   copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
>   it master.xml, or it can copy it wholesale in to master.xml.

See proposal #3 for how this is done.  We definitely want to install the
OMF file somewhere other than in the database so that the database can be
rebuilt from original OMF files if it is corrupted.

>   The former might look like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf url="file:/var/db/sk/omf/<md5>.omf">
>      </doc>
> 
>   the latter might look like
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf>
>          <omf:title>The App-name manual</omf:title>
> 
>          <!-- Other elements here -->
>        </omf>
>      </doc>
> 
>   It shouldn't matter to the end user.
> 
>   The user can now regenerate their HTML list of documents installed on
>   the system.  They might do something like
> 
>     sk-generate-contents -format html -db /var/db/master.xml > contents.html
> 
>   which would generate contents.html, which, in a browser, might look
>   something like this;
> 
>  ==========================================================================
>   Category: Unfiled
> 
>     The App-Name manual
> 
>       Include meta information from the OMF here (SK ID = MD5)
> 
>       Formats: Local HTML (split), PDF, Remote HTML (split)
>  =========================================================================

This may be useful for people who don't have a help-browser aside from
Netscape.  Although Netscape won't know how to display most docs.

>   I anticipate that there will be a system wide sk Contents List (like 
>   /var/db/sk/master.xml in my examples) that will be used to generate
>   a system wide HTML (and other formats) Contents List regularly, through
>   cron, or something similar.

The database and any exported files like this are updated each time a new
package/doc is installed, uninstalled, or updated.  We may have to use
cron if we want to have automatic processing of man and info files since
the install setups for these will not generally call sk.

>   Notice that all this works whether the end user is root or not.  If the
>   documentation has been installed somewhere under $HOME then the user
>   can adjust sk-install paths as necessary.

Yes.  This is a very nice feature.  The help browser can provide a utility
(or one could use an external tool) which allows users to easily create
OMF files to include their own docs and to modify their personal
categories list.

> Install new documentation that doesn't come with OMF
> 
>   This is very similar to the previous example.  However, instead of 
>   running sk-install-omf, the user has two choices.
> 
>    1.  Skip the step entirely.  In which case, when the contents list is
>        generated there won't be a document title, just a filename.
> 
>    2.  Run one of the helper apps I talked about earlier, that tries to
>        parse the document and make a stab at generating the OMF.  The
>        user can then clean this up hand as necessary, and run sk-install-omf
>        by hand (if they're so inclined).  They can also submit the OMF 
>        they've written back to the original author.
> 
> Generate TOC and Index
> 
>   sk-generate-toc -format html -db /var/db/master.xml
> 
>   sk-generate-index -format html -db /var/db/master.xml

As with the contents list, we could export html.  However I think most
browsers will find XML much more useful.

>   I'll hand wave over these.  Again, they should be cron'able, and probably
>   need to support various different output formats.
> 
> Removing documentation
> 
>   If the user wants to remove a single document format, but keep the 
>   information about the document, they might do
> 
>     sk-remove-doc -format pdf -id MD5
> 
>   which would remove the <docinstance> information for the PDF version of
>   the document with id == MD5.
> 
>   If they want to remove a complete document, they do
> 
>     sk-remove-doc -id MD5
> 
>   which removes the whole thing.  This does not touch the document's files
>   (such as /usr/local/share/doc/app-name/*), but it does update the Contents
>   List, and if the OMF has been stored in a separate file then it removes
>   that as well.

As above, we don't want to specify metadata (eg. the doc format) or the
id's on the command line.  The timestamps and existance of OMF files are
sufficient to determine if an entry should be updated or removed.

We could optionally pass the OMF file path or doc file path to a special
uninstall script as was in proposal #1, but this complicates things and is
unnecessary.

[SNIP]


I like a number of these ideas.  You clearly spent quite a bit of time on
this and found a number of uses and features we completely missed.  

I'm unclear on how and why we should embed contents lists inside each
other.  I'm also very scared by the idea of listing metadata on the
command line in addition to inside the OMF metadata file.  I think this
creates a lot more work for packagers and risks introducing a lot of
errors and inconsistancies in the metadata without gaining anything.  The
obvious exception is the case where an OMF file does not exist, but I
think creating the metadata and installing it should be two seperate
steps.

Dan

[Scrollkeeper-devel] system similar to ScrollKeeper

[Bob S. <bo...@sc...>]

This mail describes the ODSSI (Online Documentation System
Services Interface) system that I mentioned in an earlier
mail. I'm sending this because ODSSI has goals similar to
ScrollKeeper.  Let me be clear that this is *not* a
proposal for ScrollKeeper, nor I'm not expecting any
reviews of our system (believe me, with three years
of experience using it, I'm well aware of its
weaknesses 8^). This mail is simply a description of one
actual implementation of a similar system.  I'll present
what we did, and then describe how I think it relates to
ScrollKeeper.  You can skip down to "How this relates
to ScrollKeeper" if you don't want all the details.

The SCO Help System
-----------------------

The SCO help system consists of an HTTP server delivering
HTML and man page content.  Content can be browsed locally
or remotely using any modern browser (see
<http://uw7doc.sco.com> for an example).  Local browsing is
also done through the HTTP server, using http://localhost.
Our first release used a customized browser, but we
abandoned that approach because once the system is on a
network, people wanted to use other browsers.  So we moved
all special processing to the server side.

Where's the doc?
-----------------------

The help server runs on a dedicated port and all doc is
installed below its DocumentRoot into language subtrees.
Under each language is a flat directory structure, one
"book" per directory.

DocumentRoot
  |
  |- en
  |   |- DesktopGuide
  |   |- ModemSetup
  |   |- NetworkConfig
  |   |- PortingGuide
  |   |...
  |- fr
  |   |- DesktopGuide
  |   |- NetworkConfig
  |   |...               [not all content is translated]

Here "book" is taken to mean a collection of html files
under one directory. The directory name also serves as a
unique book ID.  A book is a coherent unit of documentation
that should stand on its own, although it may reference
other books. It can be a real multi-chapter book, or a
short topic like a HOWTO.  Most of our content shifted from
big books to small single subject booklets, more like
HOWTOs.  Most of our content is also "chunked" into many small
HTML files for quick response.

If a book has more than one chapter, the book TOC lists the
chapter TOCs, each of which in turn lists its individual html
files.  For updates, we ship a new chapter TOC if a chapter
has changed, and the installation tools can rebuild the
main TOC if we add or remove a chapter.  Or a book can
simply supply a single fixed TOC and replace that in an
update.

Sometimes we use symlinks to link in content from other
parts of the filesystem so it becomes "visible" under
the DocumentRoot of the HTTP server.   Another exception is
man pages, which are installed in their usual location
/usr/man.  Man pages are accessed through CGI to get at
them. The CGI script also lets the system match the
functionality of command line man to find a man page
(section selection, apropos, MANPATH, etc.).


Content management goals 
---------------------------

These are the goals of the help system and ODSSI:

- Serve books, smaller topics, and man pages.
- Support multiple languages, with fallback using content negotiation.
- Support additional new content, from SCO and third parties.
- Support incremental page additions, replacements, deletions.
- Uninstall content cleanly (leave no traces).
- Organize the content into subjects for presentation.
- Regenerate navigational list pages when content changes.
- Regenerate search indexes.
- Minimize the burden on content providers.
- Support content with incomplete metadata.
   
Doc installation
---------------------------

Packages are expected to install doc under the DocumentRoot
location. If that isn't feasible because of other
requirements, the package needs to symlink the content
to appear under DocumentRoot, otherwise the HTTP server
cannot reach it.  

Each package installing documentation is expected to
also install three metadata files along with its doc content.
Then the package postinstall script is expected to call three or
four help configuration scripts already resident on the system.
These four scripts comprise the Online Documentation 
System Service Interface (ODSSI):

config_book   -- configures a book
config_man    -- configures a man section
config_views  -- updates the navigational pages
config_search -- updates search indexes

Configuring a book
---------------------------

This is what config_book does for each book.
If it is called without a bookid argument, it
processes all books that need processing.

1.  Determine the list of languages to process
from the language option, or default to English.

2.  For each language, set the LANG 
environment variable (this is necessary to get
proper sorting of titles if that is called for).

3.  cd to the book directory, and read all the *.node metadata
files to assemble a list of active HTML filenames and their
Title fields.  If there are no *.node metadata files, the
fallback is to collect the list of filenames and <TITLE>
elements from within the HTML files in the directory.

4.  If the *.book metadata file permits it, rebuild the
book table of contents.

5.  Generate the file list used by the search indexer.

The idea of "active" files came from a packaging limitation
that made it impossible for an update package to delete
an obsoleted file owned by another package.  Instead, we ship
new metadata for a file with status field marked as "delete".
Then that file is excluded from further processing
and taken out of the search index.  It just hangs
around on the system until its original package
is removed.

To regenerate the book TOC, the script looks in the *.book
metadata file, which identifies the type of TOC:

alpha   - book TOC is an alphabetical list of titles.
chapter - book TOC is a list of chapter titles.
fixed   - book TOC is fixed and may not be regenerated.

An "alpha" TOC is regenerated from the sorted list of Title
fields for the active files.  This TOC style is used for
glossary and encyclopedia type material, and man sections.

A "chapter" style TOC is rebuilt using metadata from the
*.node file that identifies active chapter TOC files.  When
a chapter is updated, we ship a new chapter TOC file.  The
rebuild of the book TOC lets us add, delete, or rearrange
the sequence of chapters in the book.  The individual
chapter TOCs then handle the rest of the section
hierarchy.  Why not just ship a new book TOC?  Because a
book can be updated from more than one independent update
package, and we don't want to overwrite other 
updates.  A TOC type of "fixed" is not regenerated.

The search indexer filelist for each book is kept on the
system. When a book is reindexed, the indexer first removes
all index records for files in the old list, and then
indexes all the files in the new list.  This lets us
eliminate obsolete files from the search index without
having to delete the whole index and reindex the whole doc
set.  

Book metadata
---------------------------

There are two metadata files per book that support the
above actions:

- A *.node file provides per-file info for a collection of files.

- A *.book file provides book-level info (title, bookid, etc).

The node file has one record for each HTML file. Here are three
sample records from a node file:

MouseOverview.html:
        Title = Overview of the mouse software
        Version = 1.0
        PrintSequence = /HW_mice/CTOC-Mouse_Administration.html

AddingMouse.html:
        Title = Adding a mouse device
        Version = 1.0
        PrintSequence = /HW_mice/CTOC-Mouse_Administration.html

CTOC-Mouse_Administration.html:
        Title = Mouse administration 
        Version = 1.0
        ChapterTOCWeight = 3

The first two records are for ordinary HTML content.
They both have a "PrintSequence" field that points to
the third record, which is a chapter table of contents
file (CTOC) of which they are a part.

A book can have several *.node files, typically an original
and one or more updates. If two node files contain metadata
for the same HTML filename, then the record with the higher
version number is used.  That permits the metadata to be
updated without replacing the original node file.  If that
higher version number includes a "Status = delete" field,
then that HTML file is dropped from the active list as
described above.

The "ChapterTOCWeight" field of the CTOC record identifies
that file as a chapter TOC, and suggests where its entry
should appear in the regenerated book TOC.  The weight
fields are sorted numerically for the book TOC.
The numbers can be decimals (e.g., 3.1) to permit
a new chapter to be inserted between two existing chapters.
An update package can drop a chapter without deleting it
by including a new record with "Status = delete" for
the CTOC file.

Each CTOC file is a typical nested list HTML file
presenting the chapter section titles. But it also contains
parsable metadata for that chapter hierarchy, identifying the
sequence and section level of all the HTML section files.
That metadata makes it easy for a CGI script to assemble
a complete chapter for printing, a feature users like.
It doesn't have to parse the HTML markup to figure it out.

We don't try to regenerate chapter TOCs after independent
updates because of the hierarchy.  Independent updates
of a simple list like the book TOC chapter list are easy,
but independent updates of a complex hierarchy can lead
to a scramble.

Man sections are handled by a separate config_man script.
That script generates an alpha TOC for each man section,
scanning over all the MANPATH directories. It also extracts
NAME lines to build the man apropos database.  It can
handle man pages in three formats: man, cat, and HTML.

Configuring the site
---------------------------

The above process manages just the content within a book
directory or man section. There is another script
"config_views" to manage how each book fits into the mix of
content on the server.

In our first design, we divided up the domain of Unix OS
information into what we thought was a complete subject
hierarchy.  We would be fitting each book into that fixed
hierarchy.  It didn't work.  New categories emerged as the
industry progressed (or we discovered them).  And some
content was hard to fit into one location in the
hierarchy.  I like to use the example of networked
printers.  Does that go under "Printers" or "Networking"?
Users are likely to look under either, so it needs to be
under both.  Which meant we either repeated information in
several books, or we devised a way to reuse information.

It became clear that there are several ways to "slice"
and present a large collection of information, depending
on your point of view (developer, administrator, end user).
Certain subjects could be handled by simply pointing at
a specific book.  But others needed to draw on parts of
other books, and even selections of man pages.  

So we developed the idea of "views" as a layer on top of the
content.  A view is a listing of content drawn from various
points in the doc collection.  Each view is a flat list of
view records.  An individual view record is an HREF that
can point to another view file, a book TOC, a chapter TOC,
a man section TOC, or an individual HTML file.  As you follow
a sequence of links, you might pass down through two views,
a book TOC, a chapter TOC, and then land on your desired file. 
Another sequence might be just two views and then a man page.

You might think this is a hopeless mess, mixing categories
and content willy nilly.  But the user sees it as the
shortest path to what they are looking for (one hopes).
The views are user oriented, while the content is 
package oriented.  You can draw an analogy with
documentation that sequentially covers all the menu
options in a program.  It is highly organized and complete.
But it doesn't much help a user who needs to put together
items from four different menus to accomplish a task. 

This design permits the creation of a view hierarchy that
supports multiple presentations of the same material,
organized in different ways.  For example, we present
a system Handbook that has no chapters of its own, but is a
collection of selected chapters from many books.
You can see this in action at http://uw7doc.sco.com,
scroll down and select "Handbook" from the left frame list.

Also, by using separate view
metadata, it lets you update or ship whole new view
branches after the doc itself ships.  Administrators
could even write their own views, similar to creating a 
bookmark hierarchy.

The other major feature of the design is that the view data
is modular, not monolithic.  The complete view hierarchy is
assembled from a lot of small contributions provided by
the packages.  Each package provides information about its own
content and "hints" for how it should be presented.

View metadata
---------------------------

Each book includes a *.view metadata file that contains one
or more view records that pertain to that book's content.
A single record of view metadata can have these fields:

viewID:
      Title = hot spot text
      URL = hot spot URL
      ParentView = viewID that contains this hot spot
      [Frame = framename]
      [Class = classname]
      [Weight = numerical sort of arbitrary numbers]
      [OverviewURL = URL for intro text]
      [Version = version number for this viewID]

View records are really of two kinds: those that point to
existing HTML files (or TOCs), and those that generate
new view pages.  

viewID
   A unique string that identifies a view record.
   The string can include "/"s, so they typically
   contain some path info to make them unique.

   A piece of doc can have more than one view record
   associated with it, with different viewIDs. That
   enables the doc to show up in more than one place
   in the view hierarchy.

Title and URL
   These are used to form the hyperlink:
      <A HREF="URL">Title</A>

ParentView
   The page that contains this hyperlink.
  
Frame
   Optional field to add TARGET="framename" to the
   hyperlink.
   
Class and Weight
   Are used to group and sort view records that appear
   on the ParentView.  All view records on a view page
   with the same Class name are grouped together between
   <HR> lines.  This partitions the list into subsections.
   The null Class name appears as the first group, with
   other group appearing in alpha order of Class name.
   Within a given Class name, view records are sorted by
   the numerical weight values.  "Weightless" records float
   to the top.  Finally, all records with
   the same Class and Weight are sorted alphabetically.
   Thus if no Class or Weight information is given for any
   records on a view page, you get a straight alphabetical
   list on that view page.

OverviewURL
   For a view record that defines a new view page, this URL
   points to content that explains the view page.
   The <H1> on the generated view page is made into a
   hyperlink that points to this URL.  In a frames-based
   system, this content is also displayed in the right
   frame while the view page is displayed in the left
   frame.

Version
   Version of the view record.  If more than one record for a
   given viewID exists in different *.view files, only the
   highest versioned record is used.  This lets you
   supercede a view record without deleting the original,
   which might be in a different package.

The help system ships with a set of top level view records.
The help home page has viewID "topview", and its ParentView
is "ROOT".  No other page can have parent ROOT.  A set
of basic categories are shipped with ParentView = topview
to populate the home page with the likely suspects of
the basic installed doc set.  The rest of the hierarchy
comes from the doc packages *.view files.

View processing
---------------------------

When a doc package is installed, the postinstall script
executes config_views.  In the current version, the entire view
hierarchy is rebuilt each time.  Here is what happens:

1.  Processes English first, and then other languages.  This
assumes the English doc set is complete, and is used to
fill in the gaps in the translated views with untranslated
content.  This ensures that all languages have access to the
same set of information, even if some is in English.

2.  Set LANG for the current language (to ensure proper
sorting of titles).

3.  Run a "find" command under DocumentRoot/<LANG> to locate
all *.view files for that language, and load the
data into associative arrays.

4.  For duplicate viewID records, keep only the record with
the highest Version number.

5.  Make sure a viewID exists for every ParentView trying to
reference it.

6.  Check the URLs to make sure they exist.  Any view record
whose URL does not exist is ignored.  This ensures that all
view links work.

7.  Sort the view records by:
     parent
     class
     weight
     title 

8.  For each new parent, generate an HTML view page using
its title and writing to its URL.  If it has an OverviewURL,
make its <H1> a hyperlink to the OverviewURL.

9.  For each view record for that page, generate a <A HREF=...>
hyperlink.  Draw <HR> when Class name changes.  

10.  Repeat for other languages.


Note that the view database is not kept, only the
generated HTML view files derived from it.  The processed
view records exist only in memory until config_views
exits.  This ensures that the view data cannot get out of
synch with a persistent database.  Also, this is the only
time the view data is used, and it only takes a few minutes
to regenerate all the views.  If this data had other
purposes it would be written out to disk.

How this relates to ScrollKeeper
---------------------------------------

Like I said, this system is not a proposal for ScrollKeeper
because of differences like these:

- All doc content resides under a DocumentRoot and is served with
  an HTTP server.

- Most of the content comes from one vendor (SCO), so there
  is a lot of coordination between packages.

- Big emphasis on handling incremental updates.

- Only serving HTML and man page document types.

- No persistent database of content.

But I think there were some lessons learned that do apply to
ScrollKeeper:

1.  A predetermined complete categorization of content is
not useful or even possible.  Rather, an open framework
of general categories can be established so providers
have a place to hang their stuff.  But providers need
the ability to add subcategories.

2.  There's more than one way to organize and
present the content.  Multiple different views can
be layered on top of the content collection.  This requires
looking at the content from the point of view of the multiple
different users, not just the content provider.

3.  Minimize the requirements on providers.  They should
just need to provide standard metadata and then call a single
post-install script already available on the system.
That script would take various actions based on the
metadata.  In a corollary, minimize the demands on
system administrators, since they generally have little
time to manage documentation.

4.  The system needs to handle content that has incomplete
metadata.  Its degree of participation and integration into
the system may be reduced, but it should not be excluded.
Our system went even further and processed content with
no metadata and no calls to the postinstall scripts, but
that's because we could locate all the content under
DocumentRoot.

5.  Metadata can be versioned, so new metadata can supercede
old without installing new versions of the doc.  You can
even mark some doc as obsolete without actually removing it.

6.  The system needs to take into account native language
translations, and provide a system of fallback when some
content is not available in a given language.  

7.  Clean removal of content is very important, and harder
than you think.  The removal scripts need to clean up
databases, views, and search indexes.  We didn't pay enough
attention to this item, and most of our bug reports came
after some content was incompletely removed.

I think many of these items have already been proposed
and discussed in the design so far.  

Conclusions
-----------------
Based on my experience to date, and at the risk of stepping
on someone's toes, I suggest that ScrollKeeper
divide its efforts into two areas:

1.  Inventorying doc content on a system.

2.  Presenting that content in various ways.

The inventorying effort includes defining the OMF metadata,
creating a database to house the data, and defining an API
for accessing the data.  This effort could also include
methods for finding non-registered doc (for example,
scanning RPMs for documentation sections), and fallback
processes for discovering and filling in missing metadata
for various file types.  The goal here is to define standard
methods for recording and reading the metadata.

The presentation effort can go in many different directions
to meet many different needs.  I don't think the
ScrollKeeper project can anticipate all the uses for a
documentation database like this, nor can it develop all
applications that use it.  One site will want to use a
Verity search engine to integrate doc with their other
searchable content, while another site will want to create
a website of documentation and index it with htdig, and a
third will load all the information into their existing
object-oriented database.  If there is a standard API for
extracting information from the ScrollKeeper database, then
those applications become much easier to do.

I do think the ScrollKeeper project should develop one
or more basic presentation applications, to the degree that it
can.  This will make it useful "out of the box" and
encourage content providers to add OMF data to their packages.
It will also test the data and the API, and lead to 
incremental refinement of both.

I think the ScrollKeeper project is already doing most
of this. I would just like to see a clear distinction
between forming the database and developing the applications
from it.  It would make it easier to divide up the work,
and clarify what needs to be delivered.

Of course, this is all just my opinion.  8^)

bobs
Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Tue, Nov 21, 2000 at 10:08:42AM +0000, Laszlo Kovacs wrote:
> Nik Clayton wrote:
> > 
> > On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > > [snip]
> > > > There should be no such thing as a non-relocatable package.
> > >
> > > I guess what we are trying to do is to create support for both
> > > relocatable and non-relocatable packages. It is up to the developer and
> > > the packager to decide if and how they want to use this support.
> > 
> > There should be no such thing as a non-relocatable package.
> > 
> Are you saying that we shouldn't offer the choice I outlined above and
> we should force people to build fully relocatable packages (the
> documentation part at least) even if they don't want to?

Yes.

The idea of being told "You can only install this document on your
system if you install it _here_" is completely bizarre to me.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Tue, Nov 21, 2000 at 01:36:34PM +0000, Laszlo Kovacs wrote:
> > /var/db/master.xml (or whatever it is) should not (contrary to my message)
> > handle the grouping of documents in to categories.  By that I mean that
> > it shouldn't do
> > 
> >     <sect>
> >       <title>Apps</title>
> >         <sect>
> >           <title>Xemacs</title>
> > 
> >           <doc>...</doc>
> >           <doc>...</doc>
> >           <doc>...</doc>
> >         </sect>
> > 
> >         ...
> >     </sect>
> > 
> > Why?  Because then it's a pain in the backside to have the same document
> > appear in multiple categories, as you have to list it twice.  Instead,
> > the format should be more like:
> > 
> >    <doc docid="...">
> >      <docinstance .../>
> > 
> >      <category>Applications</category>
> >      <category>Editors</category>
> >      <category>Lisp</category>
> >    </doc>
> > 
> > This can then be arranged in to groupings by the same process that generates
> > the TOC and Index.
>
> Actually it already works in the first way you described above. And I
> don't find it hard at all to manage it:-)

I think that's going to cause problems.

Consider Postgres.  That's going to install FAQs, Manuals, man pages (as
distinct from a Manual or User Guide), README files, and so on, and so 
forth.

Under your scheme, assume that we have a FAQ, Manuals, Man page (1), Man
page (2), Man page ..., and README categories, as well as a category for
Postgres.

Now, each Postgres doc is going to appear once in the Postgres category.
Then, some of those docs are going to be duplicated in the FAQ category,
some in the README category, some in the "Man Page (1)" category, and so
on.

This going to be even worse when you install a big Gnome application,
and you want to make sure that its documents are also in the Gnome
category.

I think all this duplication is a bad thing, and would much rather see
something like

   <doc docid="...">
     <docinstance url="file:/usr/local/share/man/man1/psql.1">
     <!-- Other elements ommitted -->
     <category>Postgres</category>
     <category>man(1)</category>
   </doc>

Than

  <doc docid="...">
    <docinstance url="file:/usr/local/share/man/man1/psql.1">

    <category>Postgres</category>
  </doc>

  <doc docid="...">
    <docinstance url="file:/usr/local/share/man/man1/psql.1">

    <category>man(1)</category>
  </doc>

Or,

  <category>
    <title>Postgres</title>

    <doc id="...">
      <!-- Ommitted -->
    </doc>
  </category>

  <category>
    <title>man(1)</title>

    <doc id="...">
      <!-- Ommitted -->
    </doc>
  </category>

This is basic information normalisation theory.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Tue, Nov 21, 2000 at 01:34:42PM +0000, Laszlo Kovacs wrote:
> >   1.  Maintaining one or more lists of documentation that the user wants
> >       a TOC for (the "Contents List" from the most recent proposal)
>
> Yes, but we didn't aim for more than one.

I think we need more than one.  I explain why in a bit.

> >   4.  Maintaining a Categories list
>
> What is the difference between this and (1)?

Depends on whether or not sk enforces the list of categories that 
documentation can be in or not.  If it doesn't, then no list.  

  Pro:  Everything's very easy.

  Con:  If the end user makes a typo when choosing a category for the 
        document, sk can't warn them about it.

> Multiple content lists would probably be a lower priority.
> 
> > 
> > What sort of documentation can the Contents List going to point to?
> > 
> >   1.  Locally installed standard documentation that comes with the system,
> >       such as man pages, or Info documents.
> > 
> >       QUESTION:  Is sk going to deal with non-SGML/XML documents, such as
> >       these?
>
> It will be able to categorize them if OMF files are created for them. I
> think sk wants to categorize docs based on OMF files while the ones
> without OMF files would not be considered by it. They can be considered
> by the help browser though as it happens now with Nautilus in Gnome.

I think this is a bad idea.  If you go this route then sk becomes another
format, like info, or man, or html, and the help browsers have to deal with
this, and still need to go and handle the rest of the documentation in the
system.

I think sk has a much better chance of being adopted if it is very easy for
legacy documentation to be 'adopted' by sk, so that, at the very least, the
documentation makes it in to the Contents List.  Producing detailed OMF for
legacy documentation can be done after the document has been adopted.

DD:  Legacy documentation (with, in the worst case, just a filename) can
     be adopted by SK so that it appears in the Contents List.  The end user
     won't be able to do very much with it until there's meaningful OMF for
     the legacy docs, but it's a start, and should assist users who want to
     write OMF for legacy docs.

<snip>

> > I think it is fair to say that, when run, sk will need to construct it's
> > global Contents List by parsing multiple, smaller, contents lists.  In much
> > the same way that SGML catalog files can include one another at the moment.
>
> As I said before sk works with one Content List at moment and we did not
> think about more than one. You might want to detail how you imagine
> this.

The SA generates a contents list for the host, call it /var/db/sk/contents.xml
or something (I was calling it master.xml in the previous examples).

A regular user has also installed some documentation under $HOME, and have 
created $HOME/.sk/db/contents.xml

The user wants to create TOC and Index files, personal to them, that contain
both these lists.

Either of the following approaches should work (to generate a personal TOC);

  1.  Use multiple '-d' parameters (or -c, or whatever)

        sk-generate-toc -d /var/db/sk/contents.xml -d ~/.sk/db/contents.xml

  2.  Use an environment variable, which contains a ':' delimited list of
      files to search.

        SK_CONTENTS=/var/db/contents.xml:~/.sk/db/contents.xml sk-generate-toc

  3.  Create a meta-contents file, that uses entity inclusions to pull in the
      files.  Something like this;

      <!DOCTYPE contents PUBLIC "-//ScrollKeeper Contents List 1.0//EN" [

      <!ENTITY systemwide SYSTEM "/var/db/sk/contents.xml">
      <!ENTITY personal   SYSTEM "/home/nik/.sk/db/contents.xml">
      ]>

      <contents>
        &systemwide;

        &personal;
      </content>

      [ I've assumed here that XML can do this in the same way that you can
        with SGML.  I know more SGML than I do XML, so this might not be
        the case. ]

      And then point sk-generate-toc at this meta-contents file;

        sk-generate-toc -d meta-contents.xml

> > DD:  Contents Lists must be able to refer to other Contents Lists, and sk
> >      must have an option to chase the link to read the referred to Contents
> >      List.
> > 
> > DD:  sk must have a run time option to allow the end user to select which
> >      Contents Lists the user wants sk to process.
> > 
> > DD:  sk must have a run time option to allow the end user to select which
> >      TOC to update.
>
> Waht do you mean about TOC update? At the moment this is extracted at
> install time (from sgml only) and stored in a file. As long as the doc
> does not change, the TOC does not need to be updated. 

Joe Sysadmin runs sk-generate-toc to generate a TOC suitable for the entire
host, and puts it in /usr/local/share/doc/TOC

Joe User runs sk-generate-toc to generate a TOC that includes documentation
that they've installed documentation under $HOME.

Just so I'm clear here -- I'm not talking about the TOC that gets stored in
the Contents List.  I am assuming that the TOC will need to be extracted 
from the Contents List in other formats (such as HTML) to support environments
where KDE or GNOME are not the norm.

It would be nice to assume that everybody who is going to run sk is going
to be running a desktop environment that supports these things, but not
everybody is.  I also suspect that a lot of people are going to need little
more than a mechanism which generates an HTML ToC and Index for their 
installed documentation.

> > DD:  sk must have a run time option to allow the end user to select which
> >      Index to update.
>
> Saem as for TOC above,

As above.

> > DD:  One of the sk commands must be to generate a usable Contents List
> >      (which really means {X}HTML version, to start with) given a Contents
> >      List which is in sk internal format.
>
> The sk Content List is XML, why would we want to generate HTML? I guess
> it depends on the browser what kind of files are needed. Anyway I am
> sure our XML Content List can be easily converted to HTML.

As above.

> > Each piece of documentation (which might really consist of tens or even
> > hundreds of files) can be considered to consist of two parts.
> > 
> >   1.  The documentation itself (*.html, *.png, foo.pdf, whatever).
> > 
> >   2.  The OMF, containing the 17 (?) key pieces of meta information that
> >       sk likes to keep about the documentation.  Not all 17 pieces of
> >       information may necessarily be present.
> > 
> > Not all documentation the user wants to point to is going to have the OMF
> > information associated with it.  This is unavoidable.
> > 
> > DD:  sk must work tolerably well with minimum OMF information.  At the
> >      worst case, sk must do something useful when the only OMF information
> >      available is the document title, and a URL pointing to the document's
> >      'index' page.
> > 
> >      Actually, I take that back.  The only piece of information sk needs
> >      is the URL.  In the worst case (with 0 OMF information) the URL
> >      becomes the title, until the end user provides some OMF, or points
> >      sk at pre-existing OMF.
>
> This has to be discussed. OMF is the heart of Scrollkeeper, I am not
> sure we intend to make Scrollkeeper to deal with docs that don't have
> OMFs. But as I said before, the browser can deal with them separately.

I know.  All I'm saying is that in the worst case (when a document has been
'adopted' by SK, but with no OMF, SK should synthesise an OMF that contains
1 piece of information, the document's title, which is based on the filename.

> > DD:  sk should probably include some support applications which can parse
> >      common documentation formats (*roff man and mdoc, GNU info, HTML,
> >      LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> >      them as possible.
>
> Yes, that is a good idea.

We can call them plugins.  That's a nice, sexy, buzzwordy term :-)

> >      In some cases this is necessary to produce bare bones OMF files that
> >      will not be processed further.  In other cases, this will be to produce
> >      OMF files that will be further customised and improved by the end user.
> > 
> > DD:  sk should include some support applications that can generate TOC and
> >      Index information from a variety of source formats.
>
> This is planned for SGML, it can be done for other formats too.

More plugins :-)

> > DD:  sk should not assume that the OMF information is in the same directory
> >      as the document, nor should it assume that the location of the OMF
> >      information can be inferred based on the document's location.  The
> >      location of the OMF information must always be explicitly listed in
> >      the Contents List.
>
> Yes, this is how it works.

Thought so.

> >      The OMF information may not even be on the same host as the target
> >      document.
>
> At the moment the OMF information is local, the document might not be.
> However the current implementation supports only local docs.

That's fine.  I just want to make sure that these design decisions are 
written down somewhere.  Ideally, we then have a list of things to implement,
which can be checked off one by one.

> > DD:  It is possible for a document to change, but for the OMF information
> >      to be neglected.  sk should maintain a modification time stamp for
> >      each document and its corresponding OMF, to allow the end user to
> >      determine when that might have happened, and to correct for it.
>
> Currently we take for granted that the OMF is changed at any doc change.
> If the doc has an OMF then when the whole package is re-installed then
> the OMF changes (the timestamp, this is what we monitor).

A document won't always be installed with new OMF.  For example, if SK is
maintaining information about a remote document, but with the OMF locally.
The remote document might change, and SK needs to be able to tell when the
OMF is out of date (timestamps) with respect to the document.

> > DD:  An application may have multiple pieces of documentation associated
> >      with it.  Applications are also not the only things that have
> >      documentation, the system does (think: man page sections).  Simply
> >      storing information about a document isn't enough, you need to be
> >      able to group documents into chunks (and probably group chunks into
> >      chunks, or allow the same document to appear in multiple categories).
> >      I'm not going to discuss this in too much detail in this document, this
> >      is the reason for "4. Maintaining a Categories list" earlier.
>
> I think this is working. You can certainly install a doc in multiple
> places in the Content List.

I don't think we want to install the same document in multiple places.
Apart from ballooning the size of the Contents List, it means that there 
is greater chance that one or more of those copies will be neglected during
an up date.

Each document should only appear once in the Contents List, but SK should be
able to handle multiple categories assigned to the same document.

> > DD:  One document, multiple output formats.  The user might have installed
> >      the same document three times, once as HTML, once as PDF, and once as
> >      plain text.  The three installations are the same document, and have
> >      the same OMF associated with them.
> > 
> >      sk should treat this as one document with three formats, rather than
> >      three separate documents.
> > 
> >      [ This is probably going to be the most contentious piece of this
> >        writeup, and it might change.  In particular, it also suggests that
> >        translations of a document should be treated as the same document,
> >        just in a different language, and I'm not sure that's a good idea.
> >        Then again, it might be. ]
>
> It doesn't work like this right now, every doc needs a separate OMF
> file. However one OMF file with several doc entries each describing a
> different doc is supported. So it would be possible to use one OMF file
> for the same doc in multiple formats.

This is a pretty crucial decision to make, and I haven't seen it discussed
on the list.

Gut instinct tells me that it's a cleaner design to consider that is 1:M
mapping between documents, and instances, instead of a 1:1 mapping.

I'll try and come up with some convincing reasons for this.

> > SK Tasks
> > --------
> > 
> > There are a number of tasks that the sk end user is going to want to tell
> > sk to do.  They are;
> > 
> > Install new documentation that comes bundled with OMF

<snip>

> >   'sk-install-doc' registers the document.  The two (optional) arguments
> >   here are

<snip>

> >   This command does not physically copy the file in to place.
>
> I guess here you mean that most or all of the OMF metadata should be
> able to be passed in as a parameter in the installation script/binary.
> Good idea.

No.  At this point, no OMF information has been installed.

Installing the OMF for this document is a separate task.

[ Re MD5 ]

> md5 was ruled out before because it is too slow on Solaris. But we have
> a mechanism of assigning unique IDs by storing the assigned ones and
> then just not reassigning them again.

This is implementation specific, and not really important. 

<snip>

> The current implementation assigns different IDs for the same doc in
> different formats. 

This comes back to the earlier point, about a 1:M or 1:1 mapping.  This
needs to be decided.

<snip>

> All of this can be implemented, but I don't see it too important for the
> moment. 

I do, which is why I'm bringing it up.  I hadn't seen anything talking about
SK from a user's point of view, or the sort of tasks a user is going to 
want to carry out.  I'd like to hammer out some sort of specification that
we can try and agree on, and codeto.

> > Install new documentation that doesn't come with OMF
> > 
> >   This is very similar to the previous example.  However, instead of
> >   running sk-install-omf, the user has two choices.

<snip>

> I guess we can support this later on.

We should be able to support this out of the box.  Remember that if the
document has no OMF supplied by the end user then (IMHO) SK should synthesise
some OMF.

> > Removing documentation
> > 
> >   If the user wants to remove a single document format, but keep the
> >   information about the document, they might do
> > 
> >     sk-remove-doc -format pdf -id MD5
> > 
> >   which would remove the <docinstance> information for the PDF version of
> >   the document with id == MD5.
> > 
> >   If they want to remove a complete document, they do
> > 
> >     sk-remove-doc -id MD5
> > 
> >   which removes the whole thing.  This does not touch the document's files
> >   (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> >   List, and if the OMF has been stored in a separate file then it removes
> >   that as well.
> > 
> >   Normally, this behaviour would be carried out by pkg_delete, or whatever
> >   packaging system the user is using.
>
> This is fully automatic, if you delete the OMF file, all the related
> docs will be uninstalled.

I might not want to do that though.  I might be deleting the OMF file because
it's out of date, and I haven't got time to update it.

As I say, in the absence of OMF, SK should fake some reasonable OMF, and
keep going.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

[snip]
> 
> Second;
> 
> /var/db/master.xml (or whatever it is) should not (contrary to my message)
> handle the grouping of documents in to categories.  By that I mean that
> it shouldn't do
> 
>     <sect>
>       <title>Apps</title>
>         <sect>
>           <title>Xemacs</title>
> 
>           <doc>...</doc>
>           <doc>...</doc>
>           <doc>...</doc>
>         </sect>
> 
>         ...
>     </sect>
> 
> Why?  Because then it's a pain in the backside to have the same document
> appear in multiple categories, as you have to list it twice.  Instead,
> the format should be more like:
> 
>    <doc docid="...">
>      <docinstance .../>
> 
>      <category>Applications</category>
>      <category>Editors</category>
>      <category>Lisp</category>
>    </doc>
> 
> This can then be arranged in to groupings by the same process that generates
> the TOC and Index.
Actually it already works in the first way you described above. And I
don't find it hard at all to manage it:-)

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

> Based on the design docs on the web, what does sk have responsibility for:
> 
> [ This is my current understanding.  Please correct any misconceptions I
>   might have.  Things I flag with 'DD' are design decisions that I think
>   fall out of the model. ]
> 
>   1.  Maintaining one or more lists of documentation that the user wants
>       a TOC for (the "Contents List" from the most recent proposal)
Yes, but we didn't aim for more than one.
 
>   2.  Maintaining a TOC from the Contents List
Yes.
 
>   3.  Maintaining an Index from the Contents List
Yes.
 
>   4.  Maintaining a Categories list
What is the difference between this and (1)?
 
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
>   application will have the responsibility for handling user queries of the
>   SK TOC and Index files ]
> 
> Activities (2) and (3) require a Contents List.  Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
Multiple content lists would probably be a lower priority.

> 
> What sort of documentation can the Contents List going to point to?
> 
>   1.  Locally installed standard documentation that comes with the system,
>       such as man pages, or Info documents.
> 
>       QUESTION:  Is sk going to deal with non-SGML/XML documents, such as
>       these?
It will be able to categorize them if OMF files are created for them. I
think sk wants to categorize docs based on OMF files while the ones
without OMF files would not be considered by it. They can be considered
by the help browser though as it happens now with Nautilus in Gnome.

>   2.  Locally installed system documentation, installed by the SA team.
> 
>   3.  Locally installed documentation, installed by the user.
> 
>   4.  Remote documentation on a web site.
Yes to all.

> 
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists.  In much
> the same way that SGML catalog files can include one another at the moment.
As I said before sk works with one Content List at moment and we did not
think about more than one. You might want to detail how you imagine
this.

> 
> DD:  Contents Lists must be able to refer to other Contents Lists, and sk
>      must have an option to chase the link to read the referred to Contents
>      List.
> 
> DD:  sk must have a run time option to allow the end user to select which
>      Contents Lists the user wants sk to process.
> 
> DD:  sk must have a run time option to allow the end user to select which
>      TOC to update.
Waht do you mean about TOC update? At the moment this is extracted at
install time (from sgml only) and stored in a file. As long as the doc
does not change, the TOC does not need to be updated. 
 
> DD:  sk must have a run time option to allow the end user to select which
>      Index to update.
Saem as for TOC above,

> 
> DD:  One of the sk commands must be to generate a usable Contents List
>      (which really means {X}HTML version, to start with) given a Contents
>      List which is in sk internal format.
The sk Content List is XML, why would we want to generate HTML? I guess
it depends on the browser what kind of files are needed. Anyway I am
sure our XML Content List can be easily converted to HTML.

> 
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
> 
>   1.  The documentation itself (*.html, *.png, foo.pdf, whatever).
> 
>   2.  The OMF, containing the 17 (?) key pieces of meta information that
>       sk likes to keep about the documentation.  Not all 17 pieces of
>       information may necessarily be present.
> 
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it.  This is unavoidable.
> 
> DD:  sk must work tolerably well with minimum OMF information.  At the
>      worst case, sk must do something useful when the only OMF information
>      available is the document title, and a URL pointing to the document's
>      'index' page.
> 
>      Actually, I take that back.  The only piece of information sk needs
>      is the URL.  In the worst case (with 0 OMF information) the URL
>      becomes the title, until the end user provides some OMF, or points
>      sk at pre-existing OMF.
This has to be discussed. OMF is the heart of Scrollkeeper, I am not
sure we intend to make Scrollkeeper to deal with docs that don't have
OMFs. But as I said before, the browser can deal with them separately.
 
> DD:  sk should probably include some support applications which can parse
>      common documentation formats (*roff man and mdoc, GNU info, HTML,
>      LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
>      them as possible.
Yes, that is a good idea.

> 
>      In some cases this is necessary to produce bare bones OMF files that
>      will not be processed further.  In other cases, this will be to produce
>      OMF files that will be further customised and improved by the end user.
> 
> DD:  sk should include some support applications that can generate TOC and
>      Index information from a variety of source formats.
This is planned for SGML, it can be done for other formats too.
 
> DD:  sk should not assume that the OMF information is in the same directory
>      as the document, nor should it assume that the location of the OMF
>      information can be inferred based on the document's location.  The
>      location of the OMF information must always be explicitly listed in
>      the Contents List.
Yes, this is how it works.
 
>      The OMF information may not even be on the same host as the target
>      document.
At the moment the OMF information is local, the document might not be.
However the current implementation supports only local docs.
 
> DD:  It is possible for a document to change, but for the OMF information
>      to be neglected.  sk should maintain a modification time stamp for
>      each document and its corresponding OMF, to allow the end user to
>      determine when that might have happened, and to correct for it.
Currently we take for granted that the OMF is changed at any doc change.
If the doc has an OMF then when the whole package is re-installed then
the OMF changes (the timestamp, this is what we monitor).
 
> DD:  An application may have multiple pieces of documentation associated
>      with it.  Applications are also not the only things that have
>      documentation, the system does (think: man page sections).  Simply
>      storing information about a document isn't enough, you need to be
>      able to group documents into chunks (and probably group chunks into
>      chunks, or allow the same document to appear in multiple categories).
>      I'm not going to discuss this in too much detail in this document, this
>      is the reason for "4. Maintaining a Categories list" earlier.
I think this is working. You can certainly install a doc in multiple
places in the Content List.

> 
> DD:  One document, multiple output formats.  The user might have installed
>      the same document three times, once as HTML, once as PDF, and once as
>      plain text.  The three installations are the same document, and have
>      the same OMF associated with them.
> 
>      sk should treat this as one document with three formats, rather than
>      three separate documents.
> 
>      [ This is probably going to be the most contentious piece of this
>        writeup, and it might change.  In particular, it also suggests that
>        translations of a document should be treated as the same document,
>        just in a different language, and I'm not sure that's a good idea.
>        Then again, it might be. ]
It doesn't work like this right now, every doc needs a separate OMF
file. However one OMF file with several doc entries each describing a
different doc is supported. So it would be possible to use one OMF file
for the same doc in multiple formats.
 
> SK Tasks
> --------
> 
> There are a number of tasks that the sk end user is going to want to tell
> sk to do.  They are;
> 
> Install new documentation that comes bundled with OMF
> 
>   The user has just downloaded and installed a package that comes with one
>   or more bundled documents, and the package maintainer has thoughtfully
>   provided an OMF file for each document.
> 
>   Each document has been provided in HTML and PDF format.
> 
>   Suppose the user installs the HTML documentation first, and installs it
>   it into /usr/local/share/doc/app-name/html/, where normal file is called
>   index.html.
> 
>   [ Everywhere I say "the user runs" I really mean "The package maintainer
>     does this by means of a post-install script that they've already written.
>     It is assumed that this post-install script knows the directories that
>     documentation has been installed in, by virtue of being part of the
>     package that did the installation in the first place. ]
> 
>   The user runs
> 
>      sk-install-doc -format html-split -db /var/db/sk/master.xml \
>          /usr/local/share/doc/app-name/html/index.html
> 
>   'sk-install-doc' registers the document.  The two (optional) arguments
>   here are
> 
>      -format    The format of the installed documentation.  Might be
>                 html-split      - Bunch of small HTML files
>                 html            - One big HTML file
>                 ps              - postscript
>                 pdf             - PDF
>                 ...             - extend as necessary (rtf, txt, pdb, ...)
> 
>                 If not specified, sk can try and guess what it is.
> 
>      -db        The Contents List to update.  Uses a default if not specified.
> 
>   [ Note:  This also needs a category option, so that the end user can
>            specify which categories the document goes in to.  By default,
>            it goes in to "unfiled". ]
> 
>   This command does not physically copy the file in to place.
I guess here you mean that most or all of the OMF metadata should be
able to be passed in as a parameter in the installation script/binary.
Good idea.
 
>   sk-install-doc should generate a unique identifier for this document
>   (MD5 hash of the filename?), and display this to the user.  We probably
>   also need a standalone utility (sk-generate-id?) that, given a path,
>   generates an ID.
md5 was ruled out before because it is too slow on Solaris. But we have
a mechanism of assigning unique IDs by storing the assigned ones and
then just not reassigning them again.
 
>   At this point, /var/db/sk/master.xml looks something like this:
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   As you can see, not a lot of information in there so far.  "lang" and
>   "encoding" could also be specified on the command line, and if not
>   specified, some defaults are used.
> 
>   I think we probably also need a modification time attribute, but I haven't
>   shown that here.
> 
>   Now, the user uses sk-install-doc again, but this time they install the
>   PDF file.
> 
>      sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
>          /usr/local/share/doc/app-name/pdf/doc.pdf
> 
>   A very similar command line.  However, because this is another instance of
>   the same document, this time the user specifies the ID of the document.
>   master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Finally, the chap that wrote the application also maintains information
>   about the document on the web.  So the user decides to do the following;
> 
>     sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
>       http://www.example.com/app-name/index.html
> 
>   and master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="http://www.example.com/app-name/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Now, typically, the end user wouldn't run any of this themselves.  It
>   is the responsibility of whoever produced the package/rpm/whatever to
>   make sure that this happens.
The current implementation assigns different IDs for the same doc in
different formats. However I don't see this as a problem as when we
uninstall docs the OMF files are deleted, scrollkeeper then uninstall
all the related docs.
 
>   I understand that some people have said that that's too much work to
>   expect package maintainers to do.  I don't think that's the case.  It
>   doesn't matter if the original program's author, or the package maintainer
>   doesn't want to do it.  All it takes is someone to find this functionality
>   useful enough to spend the five minutes constructing the sk-* command
>   lines to do this, and submit it back to the maintainer.  These sorts of
>   commands are not going to change a great deal between package releases.
That sounds good.

> 
>   [ In fact, I see this as being a differentiator between operating systems.
>     SuSE (to pick an example) might not bother generating OMF files for
>     applications that don't already provide them.  But it might be a point
>     of pride for the Debian people that they don't bundle a third party
>     application until the person making the .deb file has generated an OMF
>     file for it ]
> 
>   OK, so we've told sk about the document, but we haven't pointed to
>   any of the OMF data yet.  Recall that the OMF data is the same despite
>   the different document formats.
> 
>   The end user (or their package management application) runs
> 
>      sk-install-omf -id MD5 app-name.omf
> 
>   Notice how the MD5 checksum (or whatever) is included on the command line,
>   so that sk can associate this OMF file with the document that was installed
>   earlier.
The current process is much simpler than this and it achieves the same
thing. Maybe you should also look in more detail to the current
implementation.
 
>   Now, because we've already installed the document, and sk knows where the
>   document has been installed in to, the OMF shouldn't need to contain any
>   path data.  So there's nothing for sk to need to rewrite.  sk can either
>   copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
>   it master.xml, or it can copy it wholesale in to master.xml.
> 
>   The former might look like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf url="file:/var/db/sk/omf/<md5>.omf">
>      </doc>
> 
>   the latter might look like
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf>
>          <omf:title>The App-name manual</omf:title>
> 
>          <!-- Other elements here -->
>        </omf>
>      </doc>
> 
>   It shouldn't matter to the end user.
> 
>   The user can now regenerate their HTML list of documents installed on
>   the system.  They might do something like
> 
>     sk-generate-contents -format html -db /var/db/master.xml > contents.html
> 
>   which would generate contents.html, which, in a browser, might look
>   something like this;
> 
>  ==========================================================================
>   Category: Unfiled
> 
>     The App-Name manual
> 
>       Include meta information from the OMF here (SK ID = MD5)
> 
>       Formats: Local HTML (split), PDF, Remote HTML (split)
>  =========================================================================
> 
>   I anticipate that there will be a system wide sk Contents List (like
>   /var/db/sk/master.xml in my examples) that will be used to generate
>   a system wide HTML (and other formats) Contents List regularly, through
>   cron, or something similar.
> 
>   Notice that all this works whether the end user is root or not.  If the
>   documentation has been installed somewhere under $HOME then the user
>   can adjust sk-install paths as necessary.
All of this can be implemented, but I don't see it too important for the
moment. 

> 
> Install new documentation that doesn't come with OMF
> 
>   This is very similar to the previous example.  However, instead of
>   running sk-install-omf, the user has two choices.
> 
>    1.  Skip the step entirely.  In which case, when the contents list is
>        generated there won't be a document title, just a filename.
> 
>    2.  Run one of the helper apps I talked about earlier, that tries to
>        parse the document and make a stab at generating the OMF.  The
>        user can then clean this up hand as necessary, and run sk-install-omf
>        by hand (if they're so inclined).  They can also submit the OMF
>        they've written back to the original author.
> 
> Generate TOC and Index
> 
>   sk-generate-toc -format html -db /var/db/master.xml
> 
>   sk-generate-index -format html -db /var/db/master.xml
> 
>   I'll hand wave over these.  Again, they should be cron'able, and probably
>   need to support various different output formats.
I guess we can support this later on.

> 
> Removing documentation
> 
>   If the user wants to remove a single document format, but keep the
>   information about the document, they might do
> 
>     sk-remove-doc -format pdf -id MD5
> 
>   which would remove the <docinstance> information for the PDF version of
>   the document with id == MD5.
> 
>   If they want to remove a complete document, they do
> 
>     sk-remove-doc -id MD5
> 
>   which removes the whole thing.  This does not touch the document's files
>   (such as /usr/local/share/doc/app-name/*), but it does update the Contents
>   List, and if the OMF has been stored in a separate file then it removes
>   that as well.
> 
>   Normally, this behaviour would be carried out by pkg_delete, or whatever
>   packaging system the user is using.
This is fully automatic, if you delete the OMF file, all the related
docs will be uninstalled.

> 
> Querying installed documentation
> 
>   There should probably be a command line tool to allow the user to do
>   simple queries of the Contents List.  Since the Contents List is in XML,
>   any XML aware tool will be able to handle it.  However, sometimes you
>   just want to use awk.
> 
>     sk-query -format csv
> 
>   would dump the Contents List as CSV format.
> 
>     id,name,format,path,...
> 
>   Since the data is multi-dimensional, you would probably end up with
>   multiple lines with the same id and name, but different formats and paths.
> 
>   It should be possible to query individual document IDs to retrieve
>   information about them;
> 
>     sk-query -id MD5
> 
>   and probably, given a pathname to a document, retrieve information about
>   it as well,
> 
>     sk-query -path /usr/local/share/doc/app-name/html/index.html
Good idea.

> 
> Sanity checking the Contents List
> 
>   There should be a tool to sanity check the Contents List, and warn the
>   end user of potential problems.
> 
>   Several that spring immediately to mind.
> 
>     1.  Where a <docinstance> element exists, but for which no corresponding
>         files are found on the file system.
> 
>     2.  Files for which a <docinstance> element exists which points to
>         a remote document that can no longer be found.
> 
>     3.  I glossed over it earlier, but each <docinstance> should have a
>         modtime attribute, that contains the file's modification time.
>         Part of the sanity check should ensure that the modtime attribute
>         and the files (or the remote files) modification times do match.
>         If they don't, it's likely that the OMF data and/or the TOC and Index
>         are out of date, and need to be regenerated.
> 
>   None of these are fatal errors, but they will cause the generated TOC and
>   Index to potentially be wrong.  The tools should have options to fix the
>   errors themselves (in the case of (1) and (2)).
Good idea.
 
> I hope all that makes sense.  It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer.  In particular,
> it (hopefully) dispenses with the path problem.
Yes it helps a lot. You look like having lots of experience in the area.

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

Chaps,

Two things, while I remember.

First;

On Mon, Nov 20, 2000 at 09:20:47PM +0000, Nik Clayton wrote:
> DD:  sk should include some support applications that can generate TOC and
>      Index information from a variety of source formats.

<snip>

> Generate TOC and Index
> 
>   sk-generate-toc -format html -db /var/db/master.xml
> 
>   sk-generate-index -format html -db /var/db/master.xml
> 
>   I'll hand wave over these.  Again, they should be cron'able, and probably
>   need to support various different output formats.

Done properly, this should be trivial, using XSLT.

Second;

/var/db/master.xml (or whatever it is) should not (contrary to my message)
handle the grouping of documents in to categories.  By that I mean that
it shouldn't do

    <sect>
      <title>Apps</title>
        <sect>
          <title>Xemacs</title>

          <doc>...</doc>
          <doc>...</doc>
          <doc>...</doc>
        </sect>
 
        ...
    </sect>

Why?  Because then it's a pain in the backside to have the same document 
appear in multiple categories, as you have to list it twice.  Instead,
the format should be more like:

   <doc docid="...">
     <docinstance .../>

     <category>Applications</category>
     <category>Editors</category>
     <category>Lisp</category>
   </doc>

This can then be arranged in to groupings by the same process that generates
the TOC and Index.

Supporting a hierarchy of categories is an exercise left for the reader :-)

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

Nik Clayton wrote:
> 
> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
> 
> There should be no such thing as a non-relocatable package.
> 
Are you saying that we shouldn't offer the choice I outlined above and
we should force people to build fully relocatable packages (the
documentation part at least) even if they don't want to?

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Dan M. <d-...@uc...>]

Nik,

WOW!

This is impressive.  I read about 60% of it and then decided to print it
out for a closer look and so I can keep notes.  So far it looks really
good.

Dan


On Mon, 20 Nov 2000, Nik Clayton wrote:

> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
> 
> There should be no such thing as a non-relocatable package.
> 
> By all means, allow default paths to be included, but they must always 
> be overridable at installation time by the admin.
> 
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
> 
> Based on the design docs on the web, what does sk have responsibility for:
> 
> [ This is my current understanding.  Please correct any misconceptions I
>   might have.  Things I flag with 'DD' are design decisions that I think
>   fall out of the model. ]
> 
>   1.  Maintaining one or more lists of documentation that the user wants 
>       a TOC for (the "Contents List" from the most recent proposal)
> 
>   2.  Maintaining a TOC from the Contents List
> 
>   3.  Maintaining an Index from the Contents List
> 
>   4.  Maintaining a Categories list
> 
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
>   application will have the responsibility for handling user queries of the
>   SK TOC and Index files ]
> 
> Activities (2) and (3) require a Contents List.  Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
> maintain several different TOCs and Indices from several different Contents 
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
> 
> What sort of documentation can the Contents List going to point to?
> 
>   1.  Locally installed standard documentation that comes with the system,
>       such as man pages, or Info documents.
> 
>       QUESTION:  Is sk going to deal with non-SGML/XML documents, such as
>       these?
> 
>   2.  Locally installed system documentation, installed by the SA team.
> 
>   3.  Locally installed documentation, installed by the user.
> 
>   4.  Remote documentation on a web site.
> 
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists.  In much
> the same way that SGML catalog files can include one another at the moment.
> 
> DD:  Contents Lists must be able to refer to other Contents Lists, and sk
>      must have an option to chase the link to read the referred to Contents
>      List.
> 
> DD:  sk must have a run time option to allow the end user to select which
>      Contents Lists the user wants sk to process.
> 
> DD:  sk must have a run time option to allow the end user to select which
>      TOC to update.
> 
> DD:  sk must have a run time option to allow the end user to select which
>      Index to update.
> 
> DD:  One of the sk commands must be to generate a usable Contents List 
>      (which really means {X}HTML version, to start with) given a Contents
>      List which is in sk internal format.
> 
> Each piece of documentation (which might really consist of tens or even 
> hundreds of files) can be considered to consist of two parts.
> 
>   1.  The documentation itself (*.html, *.png, foo.pdf, whatever).
> 
>   2.  The OMF, containing the 17 (?) key pieces of meta information that
>       sk likes to keep about the documentation.  Not all 17 pieces of
>       information may necessarily be present.
> 
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it.  This is unavoidable.
> 
> DD:  sk must work tolerably well with minimum OMF information.  At the
>      worst case, sk must do something useful when the only OMF information
>      available is the document title, and a URL pointing to the document's
>      'index' page.
> 
>      Actually, I take that back.  The only piece of information sk needs
>      is the URL.  In the worst case (with 0 OMF information) the URL 
>      becomes the title, until the end user provides some OMF, or points
>      sk at pre-existing OMF.
> 
> DD:  sk should probably include some support applications which can parse
>      common documentation formats (*roff man and mdoc, GNU info, HTML,
>      LinuxDoc, LaTeX, DocBook) and extract as much OMF information from 
>      them as possible.
> 
>      In some cases this is necessary to produce bare bones OMF files that
>      will not be processed further.  In other cases, this will be to produce
>      OMF files that will be further customised and improved by the end user.
> 
> DD:  sk should include some support applications that can generate TOC and
>      Index information from a variety of source formats.
> 
> DD:  sk should not assume that the OMF information is in the same directory
>      as the document, nor should it assume that the location of the OMF
>      information can be inferred based on the document's location.  The
>      location of the OMF information must always be explicitly listed in 
>      the Contents List.
> 
>      The OMF information may not even be on the same host as the target
>      document.
> 
> DD:  It is possible for a document to change, but for the OMF information
>      to be neglected.  sk should maintain a modification time stamp for
>      each document and its corresponding OMF, to allow the end user to 
>      determine when that might have happened, and to correct for it.
> 
> DD:  An application may have multiple pieces of documentation associated 
>      with it.  Applications are also not the only things that have 
>      documentation, the system does (think: man page sections).  Simply
>      storing information about a document isn't enough, you need to be
>      able to group documents into chunks (and probably group chunks into
>      chunks, or allow the same document to appear in multiple categories).  
>      I'm not going to discuss this in too much detail in this document, this
>      is the reason for "4. Maintaining a Categories list" earlier.
> 
> DD:  One document, multiple output formats.  The user might have installed
>      the same document three times, once as HTML, once as PDF, and once as
>      plain text.  The three installations are the same document, and have
>      the same OMF associated with them.
> 
>      sk should treat this as one document with three formats, rather than
>      three separate documents.     
> 
>      [ This is probably going to be the most contentious piece of this 
>        writeup, and it might change.  In particular, it also suggests that
>        translations of a document should be treated as the same document,
>        just in a different language, and I'm not sure that's a good idea.
>        Then again, it might be. ]
> 
> SK Tasks
> --------
> 
> There are a number of tasks that the sk end user is going to want to tell 
> sk to do.  They are;
> 
> Install new documentation that comes bundled with OMF
> 
>   The user has just downloaded and installed a package that comes with one
>   or more bundled documents, and the package maintainer has thoughtfully
>   provided an OMF file for each document.
> 
>   Each document has been provided in HTML and PDF format.
> 
>   Suppose the user installs the HTML documentation first, and installs it
>   it into /usr/local/share/doc/app-name/html/, where normal file is called
>   index.html.
> 
>   [ Everywhere I say "the user runs" I really mean "The package maintainer
>     does this by means of a post-install script that they've already written.
>     It is assumed that this post-install script knows the directories that
>     documentation has been installed in, by virtue of being part of the 
>     package that did the installation in the first place. ]
> 
>   The user runs 
> 
>      sk-install-doc -format html-split -db /var/db/sk/master.xml \
>          /usr/local/share/doc/app-name/html/index.html
> 
>   'sk-install-doc' registers the document.  The two (optional) arguments
>   here are
> 
>      -format    The format of the installed documentation.  Might be
>                 html-split      - Bunch of small HTML files
>                 html            - One big HTML file
>                 ps              - postscript
>                 pdf             - PDF
>                 ...             - extend as necessary (rtf, txt, pdb, ...)
> 
>                 If not specified, sk can try and guess what it is.
> 
>      -db        The Contents List to update.  Uses a default if not specified.
> 
>   [ Note:  This also needs a category option, so that the end user can
> 	   specify which categories the document goes in to.  By default, 
> 	   it goes in to "unfiled". ]
> 
>   This command does not physically copy the file in to place.
> 
>   sk-install-doc should generate a unique identifier for this document
>   (MD5 hash of the filename?), and display this to the user.  We probably
>   also need a standalone utility (sk-generate-id?) that, given a path, 
>   generates an ID.
> 
>   At this point, /var/db/sk/master.xml looks something like this:
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   As you can see, not a lot of information in there so far.  "lang" and
>   "encoding" could also be specified on the command line, and if not 
>   specified, some defaults are used.
> 
>   I think we probably also need a modification time attribute, but I haven't
>   shown that here.
> 
>   Now, the user uses sk-install-doc again, but this time they install the
>   PDF file. 
> 
>      sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
>          /usr/local/share/doc/app-name/pdf/doc.pdf
> 
>   A very similar command line.  However, because this is another instance of
>   the same document, this time the user specifies the ID of the document.
>   master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Finally, the chap that wrote the application also maintains information
>   about the document on the web.  So the user decides to do the following;
> 
>     sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
>       http://www.example.com/app-name/index.html
> 
>   and master.xml now looks like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
>                     format="pdf" lang="en_US" encoding="ISO_8859-1">
>        <docinstance url="http://www.example.com/app-name/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
>      </doc>
> 
>   Now, typically, the end user wouldn't run any of this themselves.  It
>   is the responsibility of whoever produced the package/rpm/whatever to
>   make sure that this happens.
> 
>   I understand that some people have said that that's too much work to 
>   expect package maintainers to do.  I don't think that's the case.  It 
>   doesn't matter if the original program's author, or the package maintainer
>   doesn't want to do it.  All it takes is someone to find this functionality
>   useful enough to spend the five minutes constructing the sk-* command
>   lines to do this, and submit it back to the maintainer.  These sorts of
>   commands are not going to change a great deal between package releases.
> 
>   [ In fact, I see this as being a differentiator between operating systems.
>     SuSE (to pick an example) might not bother generating OMF files for
>     applications that don't already provide them.  But it might be a point
>     of pride for the Debian people that they don't bundle a third party
>     application until the person making the .deb file has generated an OMF
>     file for it ]
> 
>   OK, so we've told sk about the document, but we haven't pointed to
>   any of the OMF data yet.  Recall that the OMF data is the same despite
>   the different document formats.
> 
>   The end user (or their package management application) runs
> 
>      sk-install-omf -id MD5 app-name.omf
> 
>   Notice how the MD5 checksum (or whatever) is included on the command line,
>   so that sk can associate this OMF file with the document that was installed
>   earlier.
> 
>   Now, because we've already installed the document, and sk knows where the
>   document has been installed in to, the OMF shouldn't need to contain any
>   path data.  So there's nothing for sk to need to rewrite.  sk can either
>   copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
>   it master.xml, or it can copy it wholesale in to master.xml.
> 
>   The former might look like this;
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf url="file:/var/db/sk/omf/<md5>.omf">
>      </doc>
> 
>   the latter might look like
> 
>      <doc docid="the MD5 checksum">
>        <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
>                     format="html-split" lang="en_US" encoding="ISO_8859-1">
> 
>        <!-- Other <docinstance> elements here -->
> 
>        <omf>
>          <omf:title>The App-name manual</omf:title>
> 
>          <!-- Other elements here -->
>        </omf>
>      </doc>
> 
>   It shouldn't matter to the end user.
> 
>   The user can now regenerate their HTML list of documents installed on
>   the system.  They might do something like
> 
>     sk-generate-contents -format html -db /var/db/master.xml > contents.html
> 
>   which would generate contents.html, which, in a browser, might look
>   something like this;
> 
>  ==========================================================================
>   Category: Unfiled
> 
>     The App-Name manual
> 
>       Include meta information from the OMF here (SK ID = MD5)
> 
>       Formats: Local HTML (split), PDF, Remote HTML (split)
>  =========================================================================
> 
>   I anticipate that there will be a system wide sk Contents List (like 
>   /var/db/sk/master.xml in my examples) that will be used to generate
>   a system wide HTML (and other formats) Contents List regularly, through
>   cron, or something similar.
> 
>   Notice that all this works whether the end user is root or not.  If the
>   documentation has been installed somewhere under $HOME then the user
>   can adjust sk-install paths as necessary.
> 
> Install new documentation that doesn't come with OMF
> 
>   This is very similar to the previous example.  However, instead of 
>   running sk-install-omf, the user has two choices.
> 
>    1.  Skip the step entirely.  In which case, when the contents list is
>        generated there won't be a document title, just a filename.
> 
>    2.  Run one of the helper apps I talked about earlier, that tries to
>        parse the document and make a stab at generating the OMF.  The
>        user can then clean this up hand as necessary, and run sk-install-omf
>        by hand (if they're so inclined).  They can also submit the OMF 
>        they've written back to the original author.
> 
> Generate TOC and Index
> 
>   sk-generate-toc -format html -db /var/db/master.xml
> 
>   sk-generate-index -format html -db /var/db/master.xml
> 
>   I'll hand wave over these.  Again, they should be cron'able, and probably
>   need to support various different output formats.
> 
> Removing documentation
> 
>   If the user wants to remove a single document format, but keep the 
>   information about the document, they might do
> 
>     sk-remove-doc -format pdf -id MD5
> 
>   which would remove the <docinstance> information for the PDF version of
>   the document with id == MD5.
> 
>   If they want to remove a complete document, they do
> 
>     sk-remove-doc -id MD5
> 
>   which removes the whole thing.  This does not touch the document's files
>   (such as /usr/local/share/doc/app-name/*), but it does update the Contents
>   List, and if the OMF has been stored in a separate file then it removes
>   that as well.
> 
>   Normally, this behaviour would be carried out by pkg_delete, or whatever
>   packaging system the user is using.
> 
> Querying installed documentation
> 
>   There should probably be a command line tool to allow the user to do
>   simple queries of the Contents List.  Since the Contents List is in XML,
>   any XML aware tool will be able to handle it.  However, sometimes you 
>   just want to use awk.
> 
>     sk-query -format csv
> 
>   would dump the Contents List as CSV format.
> 
>     id,name,format,path,...
> 
>   Since the data is multi-dimensional, you would probably end up with
>   multiple lines with the same id and name, but different formats and paths.
> 
>   It should be possible to query individual document IDs to retrieve 
>   information about them;
> 
>     sk-query -id MD5
> 
>   and probably, given a pathname to a document, retrieve information about
>   it as well,
> 
>     sk-query -path /usr/local/share/doc/app-name/html/index.html
> 
> Sanity checking the Contents List
> 
>   There should be a tool to sanity check the Contents List, and warn the
>   end user of potential problems.
> 
>   Several that spring immediately to mind.
> 
>     1.  Where a <docinstance> element exists, but for which no corresponding 
> 	files are found on the file system.
> 
>     2.  Files for which a <docinstance> element exists which points to
> 	a remote document that can no longer be found.
> 
>     3.  I glossed over it earlier, but each <docinstance> should have a
> 	modtime attribute, that contains the file's modification time.
> 	Part of the sanity check should ensure that the modtime attribute
> 	and the files (or the remote files) modification times do match.
> 	If they don't, it's likely that the OMF data and/or the TOC and Index
> 	are out of date, and need to be regenerated.
> 
>   None of these are fatal errors, but they will cause the generated TOC and
>   Index to potentially be wrong.  The tools should have options to fix the
>   errors themselves (in the case of (1) and (2)).
> 
> I hope all that makes sense.  It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer.  In particular,
> it (hopefully) dispenses with the path problem.
> 
> Before anyone asks, I'm not tied to any of the command names or the option
> names that I've used here, nor am I particularly tied to the XML examples;
> as long as we capture the content properly I don't care what the element
> names are :-)
> 
> N

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> [snip]
> > There should be no such thing as a non-relocatable package.
>
> I guess what we are trying to do is to create support for both
> relocatable and non-relocatable packages. It is up to the developer and
> the packager to decide if and how they want to use this support.

There should be no such thing as a non-relocatable package.

By all means, allow default paths to be included, but they must always 
be overridable at installation time by the admin.

Imagine an RPM that wanted to install in /opt, and wouldn't let you install
it anywhere else.

Based on the design docs on the web, what does sk have responsibility for:

[ This is my current understanding.  Please correct any misconceptions I
  might have.  Things I flag with 'DD' are design decisions that I think
  fall out of the model. ]

  1.  Maintaining one or more lists of documentation that the user wants 
      a TOC for (the "Contents List" from the most recent proposal)

  2.  Maintaining a TOC from the Contents List

  3.  Maintaining an Index from the Contents List

  4.  Maintaining a Categories list

[ Note: I've explicitly ommitted searching from this list -- AIUI, someother
  application will have the responsibility for handling user queries of the
  SK TOC and Index files ]

Activities (2) and (3) require a Contents List.  Note that a don't say "the
{TOC,Index}", I say "a {TOC,Index}".  This is because sk may be asked to 
maintain several different TOCs and Indices from several different Contents 
Lists on the same host (for example, a regular user might want to run sk
to maintain their own index and TOC of ~/my-docs).

What sort of documentation can the Contents List going to point to?

  1.  Locally installed standard documentation that comes with the system,
      such as man pages, or Info documents.

      QUESTION:  Is sk going to deal with non-SGML/XML documents, such as
      these?

  2.  Locally installed system documentation, installed by the SA team.

  3.  Locally installed documentation, installed by the user.

  4.  Remote documentation on a web site.

I think it is fair to say that, when run, sk will need to construct it's
global Contents List by parsing multiple, smaller, contents lists.  In much
the same way that SGML catalog files can include one another at the moment.

DD:  Contents Lists must be able to refer to other Contents Lists, and sk
     must have an option to chase the link to read the referred to Contents
     List.

DD:  sk must have a run time option to allow the end user to select which
     Contents Lists the user wants sk to process.

DD:  sk must have a run time option to allow the end user to select which
     TOC to update.

DD:  sk must have a run time option to allow the end user to select which
     Index to update.

DD:  One of the sk commands must be to generate a usable Contents List 
     (which really means {X}HTML version, to start with) given a Contents
     List which is in sk internal format.

Each piece of documentation (which might really consist of tens or even 
hundreds of files) can be considered to consist of two parts.

  1.  The documentation itself (*.html, *.png, foo.pdf, whatever).

  2.  The OMF, containing the 17 (?) key pieces of meta information that
      sk likes to keep about the documentation.  Not all 17 pieces of
      information may necessarily be present.

Not all documentation the user wants to point to is going to have the OMF
information associated with it.  This is unavoidable.

DD:  sk must work tolerably well with minimum OMF information.  At the
     worst case, sk must do something useful when the only OMF information
     available is the document title, and a URL pointing to the document's
     'index' page.

     Actually, I take that back.  The only piece of information sk needs
     is the URL.  In the worst case (with 0 OMF information) the URL 
     becomes the title, until the end user provides some OMF, or points
     sk at pre-existing OMF.

DD:  sk should probably include some support applications which can parse
     common documentation formats (*roff man and mdoc, GNU info, HTML,
     LinuxDoc, LaTeX, DocBook) and extract as much OMF information from 
     them as possible.

     In some cases this is necessary to produce bare bones OMF files that
     will not be processed further.  In other cases, this will be to produce
     OMF files that will be further customised and improved by the end user.

DD:  sk should include some support applications that can generate TOC and
     Index information from a variety of source formats.

DD:  sk should not assume that the OMF information is in the same directory
     as the document, nor should it assume that the location of the OMF
     information can be inferred based on the document's location.  The
     location of the OMF information must always be explicitly listed in 
     the Contents List.

     The OMF information may not even be on the same host as the target
     document.

DD:  It is possible for a document to change, but for the OMF information
     to be neglected.  sk should maintain a modification time stamp for
     each document and its corresponding OMF, to allow the end user to 
     determine when that might have happened, and to correct for it.

DD:  An application may have multiple pieces of documentation associated 
     with it.  Applications are also not the only things that have 
     documentation, the system does (think: man page sections).  Simply
     storing information about a document isn't enough, you need to be
     able to group documents into chunks (and probably group chunks into
     chunks, or allow the same document to appear in multiple categories).  
     I'm not going to discuss this in too much detail in this document, this
     is the reason for "4. Maintaining a Categories list" earlier.

DD:  One document, multiple output formats.  The user might have installed
     the same document three times, once as HTML, once as PDF, and once as
     plain text.  The three installations are the same document, and have
     the same OMF associated with them.

     sk should treat this as one document with three formats, rather than
     three separate documents.     

     [ This is probably going to be the most contentious piece of this 
       writeup, and it might change.  In particular, it also suggests that
       translations of a document should be treated as the same document,
       just in a different language, and I'm not sure that's a good idea.
       Then again, it might be. ]

SK Tasks
--------

There are a number of tasks that the sk end user is going to want to tell 
sk to do.  They are;

Install new documentation that comes bundled with OMF

  The user has just downloaded and installed a package that comes with one
  or more bundled documents, and the package maintainer has thoughtfully
  provided an OMF file for each document.

  Each document has been provided in HTML and PDF format.

  Suppose the user installs the HTML documentation first, and installs it
  it into /usr/local/share/doc/app-name/html/, where normal file is called
  index.html.

  [ Everywhere I say "the user runs" I really mean "The package maintainer
    does this by means of a post-install script that they've already written.
    It is assumed that this post-install script knows the directories that
    documentation has been installed in, by virtue of being part of the 
    package that did the installation in the first place. ]

  The user runs 

     sk-install-doc -format html-split -db /var/db/sk/master.xml \
         /usr/local/share/doc/app-name/html/index.html

  'sk-install-doc' registers the document.  The two (optional) arguments
  here are

     -format    The format of the installed documentation.  Might be
                html-split      - Bunch of small HTML files
                html            - One big HTML file
                ps              - postscript
                pdf             - PDF
                ...             - extend as necessary (rtf, txt, pdb, ...)

                If not specified, sk can try and guess what it is.

     -db        The Contents List to update.  Uses a default if not specified.

  [ Note:  This also needs a category option, so that the end user can
	   specify which categories the document goes in to.  By default, 
	   it goes in to "unfiled". ]

  This command does not physically copy the file in to place.

  sk-install-doc should generate a unique identifier for this document
  (MD5 hash of the filename?), and display this to the user.  We probably
  also need a standalone utility (sk-generate-id?) that, given a path, 
  generates an ID.

  At this point, /var/db/sk/master.xml looks something like this:

     <doc docid="the MD5 checksum">
       <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">
     </doc>

  As you can see, not a lot of information in there so far.  "lang" and
  "encoding" could also be specified on the command line, and if not 
  specified, some defaults are used.

  I think we probably also need a modification time attribute, but I haven't
  shown that here.

  Now, the user uses sk-install-doc again, but this time they install the
  PDF file. 

     sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
         /usr/local/share/doc/app-name/pdf/doc.pdf

  A very similar command line.  However, because this is another instance of
  the same document, this time the user specifies the ID of the document.
  master.xml now looks like this;

     <doc docid="the MD5 checksum">
       <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">
       <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
                    format="pdf" lang="en_US" encoding="ISO_8859-1">
     </doc>

  Finally, the chap that wrote the application also maintains information
  about the document on the web.  So the user decides to do the following;

    sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
      http://www.example.com/app-name/index.html

  and master.xml now looks like this;

     <doc docid="the MD5 checksum">
       <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">
       <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
                    format="pdf" lang="en_US" encoding="ISO_8859-1">
       <docinstance url="http://www.example.com/app-name/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">
     </doc>

  Now, typically, the end user wouldn't run any of this themselves.  It
  is the responsibility of whoever produced the package/rpm/whatever to
  make sure that this happens.

  I understand that some people have said that that's too much work to 
  expect package maintainers to do.  I don't think that's the case.  It 
  doesn't matter if the original program's author, or the package maintainer
  doesn't want to do it.  All it takes is someone to find this functionality
  useful enough to spend the five minutes constructing the sk-* command
  lines to do this, and submit it back to the maintainer.  These sorts of
  commands are not going to change a great deal between package releases.

  [ In fact, I see this as being a differentiator between operating systems.
    SuSE (to pick an example) might not bother generating OMF files for
    applications that don't already provide them.  But it might be a point
    of pride for the Debian people that they don't bundle a third party
    application until the person making the .deb file has generated an OMF
    file for it ]

  OK, so we've told sk about the document, but we haven't pointed to
  any of the OMF data yet.  Recall that the OMF data is the same despite
  the different document formats.

  The end user (or their package management application) runs

     sk-install-omf -id MD5 app-name.omf

  Notice how the MD5 checksum (or whatever) is included on the command line,
  so that sk can associate this OMF file with the document that was installed
  earlier.

  Now, because we've already installed the document, and sk knows where the
  document has been installed in to, the OMF shouldn't need to contain any
  path data.  So there's nothing for sk to need to rewrite.  sk can either
  copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
  it master.xml, or it can copy it wholesale in to master.xml.

  The former might look like this;

     <doc docid="the MD5 checksum">
       <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">

       <!-- Other <docinstance> elements here -->

       <omf url="file:/var/db/sk/omf/<md5>.omf">
     </doc>

  the latter might look like

     <doc docid="the MD5 checksum">
       <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
                    format="html-split" lang="en_US" encoding="ISO_8859-1">

       <!-- Other <docinstance> elements here -->

       <omf>
         <omf:title>The App-name manual</omf:title>

         <!-- Other elements here -->
       </omf>
     </doc>

  It shouldn't matter to the end user.

  The user can now regenerate their HTML list of documents installed on
  the system.  They might do something like

    sk-generate-contents -format html -db /var/db/master.xml > contents.html

  which would generate contents.html, which, in a browser, might look
  something like this;

 ==========================================================================
  Category: Unfiled

    The App-Name manual

      Include meta information from the OMF here (SK ID = MD5)

      Formats: Local HTML (split), PDF, Remote HTML (split)
 =========================================================================

  I anticipate that there will be a system wide sk Contents List (like 
  /var/db/sk/master.xml in my examples) that will be used to generate
  a system wide HTML (and other formats) Contents List regularly, through
  cron, or something similar.

  Notice that all this works whether the end user is root or not.  If the
  documentation has been installed somewhere under $HOME then the user
  can adjust sk-install paths as necessary.

Install new documentation that doesn't come with OMF

  This is very similar to the previous example.  However, instead of 
  running sk-install-omf, the user has two choices.

   1.  Skip the step entirely.  In which case, when the contents list is
       generated there won't be a document title, just a filename.

   2.  Run one of the helper apps I talked about earlier, that tries to
       parse the document and make a stab at generating the OMF.  The
       user can then clean this up hand as necessary, and run sk-install-omf
       by hand (if they're so inclined).  They can also submit the OMF 
       they've written back to the original author.

Generate TOC and Index

  sk-generate-toc -format html -db /var/db/master.xml

  sk-generate-index -format html -db /var/db/master.xml

  I'll hand wave over these.  Again, they should be cron'able, and probably
  need to support various different output formats.

Removing documentation

  If the user wants to remove a single document format, but keep the 
  information about the document, they might do

    sk-remove-doc -format pdf -id MD5

  which would remove the <docinstance> information for the PDF version of
  the document with id == MD5.

  If they want to remove a complete document, they do

    sk-remove-doc -id MD5

  which removes the whole thing.  This does not touch the document's files
  (such as /usr/local/share/doc/app-name/*), but it does update the Contents
  List, and if the OMF has been stored in a separate file then it removes
  that as well.

  Normally, this behaviour would be carried out by pkg_delete, or whatever
  packaging system the user is using.

Querying installed documentation

  There should probably be a command line tool to allow the user to do
  simple queries of the Contents List.  Since the Contents List is in XML,
  any XML aware tool will be able to handle it.  However, sometimes you 
  just want to use awk.

    sk-query -format csv

  would dump the Contents List as CSV format.

    id,name,format,path,...

  Since the data is multi-dimensional, you would probably end up with
  multiple lines with the same id and name, but different formats and paths.

  It should be possible to query individual document IDs to retrieve 
  information about them;

    sk-query -id MD5

  and probably, given a pathname to a document, retrieve information about
  it as well,

    sk-query -path /usr/local/share/doc/app-name/html/index.html

Sanity checking the Contents List

  There should be a tool to sanity check the Contents List, and warn the
  end user of potential problems.

  Several that spring immediately to mind.

    1.  Where a <docinstance> element exists, but for which no corresponding 
	files are found on the file system.

    2.  Files for which a <docinstance> element exists which points to
	a remote document that can no longer be found.

    3.  I glossed over it earlier, but each <docinstance> should have a
	modtime attribute, that contains the file's modification time.
	Part of the sanity check should ensure that the modtime attribute
	and the files (or the remote files) modification times do match.
	If they don't, it's likely that the OMF data and/or the TOC and Index
	are out of date, and need to be regenerated.

  None of these are fatal errors, but they will cause the generated TOC and
  Index to potentially be wrong.  The tools should have options to fix the
  errors themselves (in the case of (1) and (2)).

I hope all that makes sense.  It's not too different from what's gone before,
but I hope the extra verbiage makes things a little clearer.  In particular,
it (hopefully) dispenses with the path problem.

Before anyone asks, I'm not tied to any of the command names or the option
names that I've used here, nor am I particularly tied to the XML examples;
as long as we capture the content properly I don't care what the element
names are :-)

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Laszlo K. <las...@su...>]

[snip]

> There should be no such thing as a non-relocatable package.
I guess what we are trying to do is to create support for both
relocatable and non-relocatable packages. It is up to the developer and
the packager to decide if and how they want to use this support.

Laszlo