Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Dan M. <d-...@uc...> - 2000-11-21 22:39:33
|
On Mon, 20 Nov 2000, Nik Clayton wrote:
> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
>
> There should be no such thing as a non-relocatable package.
Both relocatable and non-relocatable packages exist, so scrollkeeper
should support them both in the simplest possible way.
> By all means, allow default paths to be included, but they must always
> be overridable at installation time by the admin.
>
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
>
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
The current way things are designed is that it will have one contents list
for the system. Of course you can always create another one somewhere
else using the -p flag, but generally they would contain the same
information and only the default one will get updated each time a document
is installed. By specifying which OMF directories you want to include,
these databases could be made to differ. We don't currently have a
mechanism for supporting or specifying multiple OMF directories, but that
would be fairly straightforward and is planned. This will make it very
simple for users to have their own local databases which vary from the
system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
which add to or over-ride the system defaults.
It is not clear how and why one would want to use multiple contents lists
except for individual users and/or hand customization. The problem is
that somebody has to specify what each database contains. So it works for
users, because they can hand-tweak the contents. It doesn't work very
well for system-wide databases, since everything must be specified by the
packager. So while you could imagine GNOME documents being kept in one
list and KDE documents being kept in another list, this doesn't really
achieve anything. The information which would be passed on the command
line to sk to indicate which list the doc goes in is really metadata which
belongs in the OMF. OMF and XML provide everything we need to have one
database and understand the scope of each piece. (eg. whether it is a
GNOME doc or KDE doc.)
Even in my example of users having their own configuration, we are never
merging contents list. In this case the user's personal database is built
directly from OMF files just like the system-wide database. Doing it
otherwise would make it almost impossible for a user to over-ride the
categorization of the system-wide database, over-ride or remove particular
entries in the system-wide database, or merge a doc into a part of the
database which is typically in the system-wide database. Plus it is just
harder to do and messy.
Am I overlooking some particularly good reason to try to merge contents
list?
What is the advantage of breaking the main database into pieces?
> 2. Maintaining a TOC from the Contents List
>
> 3. Maintaining an Index from the Contents List
>
> 4. Maintaining a Categories list
Well, this is somewhat debatable and worthy of discussion. Somebody has
to provide and maintain the categories list. On one side, we can just
define it as part of OMF/sk and install a static list with sk. Then no
maintenance is necessary. Or, we may consider having this specified by a
configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
sys admin could drop in a replacement if they want. For example you could
chop out all the HOWTO's and man pages if you know newbies who never touch
a command line will be using your machine.
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
sk should export searching functionality. Basic metadata searches can be
implemented from scratch in sk. We may want to rely on an external engine
for more complex searches potentially, but the help browser shouldn't have
to know or care about that. It should interface directly with sk.
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
For a given database, I agree. Eventually, I see the system having one
database (with one contents list) and possibly each user having one. Is
this what you have in mind? If you think we want more than one contents
list in the system's database, please explain why.
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
Perhaps. I think we probably do want to include this information, using
automagically generated (and therefore incomplete) OMF metadata as you
suggest below. For man and info, we would have to update this information
off of cron unfortunately. So there will be an unavoidable delay between
when a document is installed and when it is registered with sk.
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
Definitely. It should support arbitrary doc formats. For some document
formats (esp. DocBook/XML), it can extract some nice info like TOC and
Index. In fact, it will eventually be able to extract the whole OMF from
DocBook/XML docs, as Norman Walsh agreed to include any missing tags.
> 2. Locally installed system documentation, installed by the SA team.
SA =? sys admin
> 3. Locally installed documentation, installed by the user.
Yes. Although like much of this discussion, this feature will come in a
post-0.1 release.
> 4. Remote documentation on a web site.
Yes.
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
Why? It sounds very nice and wonderful. However, this adds a layer of
complexity without any obvious benefits.
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
(as above)
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
Yes. If we have seperate databases under $localstatedir and $HOME, the
user should be able to (through the help browser configuration
dialog) configure which database to use.
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
The databases are stored together. So, this should be specified at the
same time as the contents list part of the database. I don't think I want
my contents list coming from one database and my toc from another.
(Or am I misunderstanding you? I'm not sure what you mean by "update" a
TOC.)
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
(as above)
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
Everything is XML right now, so this is not an issue. If we decide
another internal format is better we should still export XML to the
browser. As for HTML, this is possible and may be a nice interface for
people who want to create a very minimalistic web browser.
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
Yes. We will have to look closely at how sk handles various pieces of
data missing.
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
Good idea.
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
Yes :)
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
yes.
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
>
> The OMF information may not even be on the same host as the target
> document.
Yes. (This is in the existing design.)
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
For the current version of sk, which does not yet allow for remote
documents, only the OMF file's timestamp is watched. This is because if
somebody edits the doc and not the OMF, then it is their fault for not
updating the OMF, not sk's. In reality whenever you install a new version
of a package, even if the OMF is not updated (as it should), it still
updates the old version so it gets a new timestamp and the new doc gets
registered despite any negligence on the part of the maintainer to update
the OMF (or the doc for that matter;).
This mainly becomes an issue for:
1) remote docs
2) docs which users write and modify themselves.
So we may indeed need to start tracking the doc timestamps when we support
these features. We have to be careful about constantly checking the
timestamps on a bunch of docs which are spread around the internet
though. There will be many cases where this task cannot complete, or else
takes an unacceptable amount of time.
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
That is why we have a contents list and categories file. This is all
specified by OMF metadata.
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
No. These definitely do not have the same OMF, unless they are identical
copies. The OMF file specifies the document format in FORMAT.DTD and
FORMAT.MIME. The docs also have different IDENTIFIER's.
(We may decide to expand this a little bit. FORMAT.STYLESHEET may be
nice, as it would allow the doc authors to dictate the appearance of their
doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
icon, etc.)
> sk should treat this as one document with three formats, rather than
> three separate documents.
If you look in the ldp-meta archives from several months ago, I proposed
we add a document id to the OMF. This would allow us to identify if two
documents are "The-Same". I call two documents "The-Same" if they are the
same document only different versions, languages, formats, etc. If two
documents have the same title, they are not necessarily
"The-Same". Likewise, if I change the title of my document at some point
in time, I can have two docs which are "The-Same" but not having the same
title.
Obviously this sort of information will be very nice to have. You can do
things like "Get me this document in language xxx." or "Show me the latest
version of this document on the Net." or "Show me the previous version of
this document from the Net."
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
I guess we were thinking along similar lines :)
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
Why do we specify the format on the command line? The FORMAT is specified
in the OMF already. The packager should just install and register the
metadata instead of putting it on the command line.
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
I don't think we really want this. One could make an argument that if no
OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
has a bunch of arguments for each element in the OMF. However I'm not
sure if we want to merge all this into the same command that registers the
document with sk.
> -db The Contents List to update. Uses a default if not specified.
Here I assume (hope?;) that you mean the database directory and not the
contents list. If so, we already do this with the "-p" flag.
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
Here we go, implementing the OMF on the command-line ;)
We should put all this metadata in the OMF metadata file.
> This command does not physically copy the file in to place.
>
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
Right now we just number them, which guarantees a unique identifier. We
used to have functions similar to the latter one, but decided to removed
them. This was because we should not encourage the help browser to
actually use these id's since they may be modified when applications are
uninstalled or installed which could break a running help browser which
doesn't realize these changes have occured.
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
I think this file should hold information taken from the OMF file, as
described in the previous proposals. Duplicating OMF information with
command-line information is redundant, puts more burden on the packager,
and increases the chance of mistakes and conflicts occuring.
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
The modification time is stored in a seperate file which keeps track of
which documents are in the database and is used for database maintenance,
updating, installing, uninstalling, etc. The contents list does not care
about the modification time.
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
>
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
Hopefully we agree that sk should be as simple and robust for the packager
and distributor as possible. This means the least amount of typing with
the smallest possible learning curve, with good fault tolerance.
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
I'd rather Suse and Debian distinguish themselves in other ways than
seeing which one can dedicate more man-hours to making a complicated
document registration system work properly ;) Perhaps they can spend this
time improving sk or some other nifty application.
And unfortunately, it isn't a matter of who does a better job but a matter
of whether sk gets adopted or not. If it is difficult to use, it will not
succeed. If it is adopted, the simpler it is the more packages that will
support it and the more useful it will be to the user.
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
Actually they are different OMF files which have different IDENTIFIER's
and FORMAT's.
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
Having the package determine the id would likely cause problems, but can
safely be omitted since sk can manage the id's itself.
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
See proposal #3 for how this is done. We definitely want to install the
OMF file somewhere other than in the database so that the database can be
rebuilt from original OMF files if it is corrupted.
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
This may be useful for people who don't have a help-browser aside from
Netscape. Although Netscape won't know how to display most docs.
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
The database and any exported files like this are updated each time a new
package/doc is installed, uninstalled, or updated. We may have to use
cron if we want to have automatic processing of man and info files since
the install setups for these will not generally call sk.
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
Yes. This is a very nice feature. The help browser can provide a utility
(or one could use an external tool) which allows users to easily create
OMF files to include their own docs and to modify their personal
categories list.
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
As with the contents list, we could export html. However I think most
browsers will find XML much more useful.
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
As above, we don't want to specify metadata (eg. the doc format) or the
id's on the command line. The timestamps and existance of OMF files are
sufficient to determine if an entry should be updated or removed.
We could optionally pass the OMF file path or doc file path to a special
uninstall script as was in proposal #1, but this complicates things and is
unnecessary.
[SNIP]
I like a number of these ideas. You clearly spent quite a bit of time on
this and found a number of uses and features we completely missed.
I'm unclear on how and why we should embed contents lists inside each
other. I'm also very scared by the idea of listing metadata on the
command line in addition to inside the OMF metadata file. I think this
creates a lot more work for packagers and risks introducing a lot of
errors and inconsistancies in the metadata without gaining anything. The
obvious exception is the case where an OMF file does not exist, but I
think creating the metadata and installing it should be two seperate
steps.
Dan
|