Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Nik C. <ni...@no...> - 2000-11-20 21:21:43
|
On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> [snip]
> > There should be no such thing as a non-relocatable package.
>
> I guess what we are trying to do is to create support for both
> relocatable and non-relocatable packages. It is up to the developer and
> the packager to decide if and how they want to use this support.
There should be no such thing as a non-relocatable package.
By all means, allow default paths to be included, but they must always
be overridable at installation time by the admin.
Imagine an RPM that wanted to install in /opt, and wouldn't let you install
it anywhere else.
Based on the design docs on the web, what does sk have responsibility for:
[ This is my current understanding. Please correct any misconceptions I
might have. Things I flag with 'DD' are design decisions that I think
fall out of the model. ]
1. Maintaining one or more lists of documentation that the user wants
a TOC for (the "Contents List" from the most recent proposal)
2. Maintaining a TOC from the Contents List
3. Maintaining an Index from the Contents List
4. Maintaining a Categories list
[ Note: I've explicitly ommitted searching from this list -- AIUI, someother
application will have the responsibility for handling user queries of the
SK TOC and Index files ]
Activities (2) and (3) require a Contents List. Note that a don't say "the
{TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
maintain several different TOCs and Indices from several different Contents
Lists on the same host (for example, a regular user might want to run sk
to maintain their own index and TOC of ~/my-docs).
What sort of documentation can the Contents List going to point to?
1. Locally installed standard documentation that comes with the system,
such as man pages, or Info documents.
QUESTION: Is sk going to deal with non-SGML/XML documents, such as
these?
2. Locally installed system documentation, installed by the SA team.
3. Locally installed documentation, installed by the user.
4. Remote documentation on a web site.
I think it is fair to say that, when run, sk will need to construct it's
global Contents List by parsing multiple, smaller, contents lists. In much
the same way that SGML catalog files can include one another at the moment.
DD: Contents Lists must be able to refer to other Contents Lists, and sk
must have an option to chase the link to read the referred to Contents
List.
DD: sk must have a run time option to allow the end user to select which
Contents Lists the user wants sk to process.
DD: sk must have a run time option to allow the end user to select which
TOC to update.
DD: sk must have a run time option to allow the end user to select which
Index to update.
DD: One of the sk commands must be to generate a usable Contents List
(which really means {X}HTML version, to start with) given a Contents
List which is in sk internal format.
Each piece of documentation (which might really consist of tens or even
hundreds of files) can be considered to consist of two parts.
1. The documentation itself (*.html, *.png, foo.pdf, whatever).
2. The OMF, containing the 17 (?) key pieces of meta information that
sk likes to keep about the documentation. Not all 17 pieces of
information may necessarily be present.
Not all documentation the user wants to point to is going to have the OMF
information associated with it. This is unavoidable.
DD: sk must work tolerably well with minimum OMF information. At the
worst case, sk must do something useful when the only OMF information
available is the document title, and a URL pointing to the document's
'index' page.
Actually, I take that back. The only piece of information sk needs
is the URL. In the worst case (with 0 OMF information) the URL
becomes the title, until the end user provides some OMF, or points
sk at pre-existing OMF.
DD: sk should probably include some support applications which can parse
common documentation formats (*roff man and mdoc, GNU info, HTML,
LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
them as possible.
In some cases this is necessary to produce bare bones OMF files that
will not be processed further. In other cases, this will be to produce
OMF files that will be further customised and improved by the end user.
DD: sk should include some support applications that can generate TOC and
Index information from a variety of source formats.
DD: sk should not assume that the OMF information is in the same directory
as the document, nor should it assume that the location of the OMF
information can be inferred based on the document's location. The
location of the OMF information must always be explicitly listed in
the Contents List.
The OMF information may not even be on the same host as the target
document.
DD: It is possible for a document to change, but for the OMF information
to be neglected. sk should maintain a modification time stamp for
each document and its corresponding OMF, to allow the end user to
determine when that might have happened, and to correct for it.
DD: An application may have multiple pieces of documentation associated
with it. Applications are also not the only things that have
documentation, the system does (think: man page sections). Simply
storing information about a document isn't enough, you need to be
able to group documents into chunks (and probably group chunks into
chunks, or allow the same document to appear in multiple categories).
I'm not going to discuss this in too much detail in this document, this
is the reason for "4. Maintaining a Categories list" earlier.
DD: One document, multiple output formats. The user might have installed
the same document three times, once as HTML, once as PDF, and once as
plain text. The three installations are the same document, and have
the same OMF associated with them.
sk should treat this as one document with three formats, rather than
three separate documents.
[ This is probably going to be the most contentious piece of this
writeup, and it might change. In particular, it also suggests that
translations of a document should be treated as the same document,
just in a different language, and I'm not sure that's a good idea.
Then again, it might be. ]
SK Tasks
--------
There are a number of tasks that the sk end user is going to want to tell
sk to do. They are;
Install new documentation that comes bundled with OMF
The user has just downloaded and installed a package that comes with one
or more bundled documents, and the package maintainer has thoughtfully
provided an OMF file for each document.
Each document has been provided in HTML and PDF format.
Suppose the user installs the HTML documentation first, and installs it
it into /usr/local/share/doc/app-name/html/, where normal file is called
index.html.
[ Everywhere I say "the user runs" I really mean "The package maintainer
does this by means of a post-install script that they've already written.
It is assumed that this post-install script knows the directories that
documentation has been installed in, by virtue of being part of the
package that did the installation in the first place. ]
The user runs
sk-install-doc -format html-split -db /var/db/sk/master.xml \
/usr/local/share/doc/app-name/html/index.html
'sk-install-doc' registers the document. The two (optional) arguments
here are
-format The format of the installed documentation. Might be
html-split - Bunch of small HTML files
html - One big HTML file
ps - postscript
pdf - PDF
... - extend as necessary (rtf, txt, pdb, ...)
If not specified, sk can try and guess what it is.
-db The Contents List to update. Uses a default if not specified.
[ Note: This also needs a category option, so that the end user can
specify which categories the document goes in to. By default,
it goes in to "unfiled". ]
This command does not physically copy the file in to place.
sk-install-doc should generate a unique identifier for this document
(MD5 hash of the filename?), and display this to the user. We probably
also need a standalone utility (sk-generate-id?) that, given a path,
generates an ID.
At this point, /var/db/sk/master.xml looks something like this:
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
</doc>
As you can see, not a lot of information in there so far. "lang" and
"encoding" could also be specified on the command line, and if not
specified, some defaults are used.
I think we probably also need a modification time attribute, but I haven't
shown that here.
Now, the user uses sk-install-doc again, but this time they install the
PDF file.
sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
/usr/local/share/doc/app-name/pdf/doc.pdf
A very similar command line. However, because this is another instance of
the same document, this time the user specifies the ID of the document.
master.xml now looks like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
format="pdf" lang="en_US" encoding="ISO_8859-1">
</doc>
Finally, the chap that wrote the application also maintains information
about the document on the web. So the user decides to do the following;
sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
http://www.example.com/app-name/index.html
and master.xml now looks like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
format="pdf" lang="en_US" encoding="ISO_8859-1">
<docinstance url="http://www.example.com/app-name/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
</doc>
Now, typically, the end user wouldn't run any of this themselves. It
is the responsibility of whoever produced the package/rpm/whatever to
make sure that this happens.
I understand that some people have said that that's too much work to
expect package maintainers to do. I don't think that's the case. It
doesn't matter if the original program's author, or the package maintainer
doesn't want to do it. All it takes is someone to find this functionality
useful enough to spend the five minutes constructing the sk-* command
lines to do this, and submit it back to the maintainer. These sorts of
commands are not going to change a great deal between package releases.
[ In fact, I see this as being a differentiator between operating systems.
SuSE (to pick an example) might not bother generating OMF files for
applications that don't already provide them. But it might be a point
of pride for the Debian people that they don't bundle a third party
application until the person making the .deb file has generated an OMF
file for it ]
OK, so we've told sk about the document, but we haven't pointed to
any of the OMF data yet. Recall that the OMF data is the same despite
the different document formats.
The end user (or their package management application) runs
sk-install-omf -id MD5 app-name.omf
Notice how the MD5 checksum (or whatever) is included on the command line,
so that sk can associate this OMF file with the document that was installed
earlier.
Now, because we've already installed the document, and sk knows where the
document has been installed in to, the OMF shouldn't need to contain any
path data. So there's nothing for sk to need to rewrite. sk can either
copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
it master.xml, or it can copy it wholesale in to master.xml.
The former might look like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<!-- Other <docinstance> elements here -->
<omf url="file:/var/db/sk/omf/<md5>.omf">
</doc>
the latter might look like
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<!-- Other <docinstance> elements here -->
<omf>
<omf:title>The App-name manual</omf:title>
<!-- Other elements here -->
</omf>
</doc>
It shouldn't matter to the end user.
The user can now regenerate their HTML list of documents installed on
the system. They might do something like
sk-generate-contents -format html -db /var/db/master.xml > contents.html
which would generate contents.html, which, in a browser, might look
something like this;
==========================================================================
Category: Unfiled
The App-Name manual
Include meta information from the OMF here (SK ID = MD5)
Formats: Local HTML (split), PDF, Remote HTML (split)
=========================================================================
I anticipate that there will be a system wide sk Contents List (like
/var/db/sk/master.xml in my examples) that will be used to generate
a system wide HTML (and other formats) Contents List regularly, through
cron, or something similar.
Notice that all this works whether the end user is root or not. If the
documentation has been installed somewhere under $HOME then the user
can adjust sk-install paths as necessary.
Install new documentation that doesn't come with OMF
This is very similar to the previous example. However, instead of
running sk-install-omf, the user has two choices.
1. Skip the step entirely. In which case, when the contents list is
generated there won't be a document title, just a filename.
2. Run one of the helper apps I talked about earlier, that tries to
parse the document and make a stab at generating the OMF. The
user can then clean this up hand as necessary, and run sk-install-omf
by hand (if they're so inclined). They can also submit the OMF
they've written back to the original author.
Generate TOC and Index
sk-generate-toc -format html -db /var/db/master.xml
sk-generate-index -format html -db /var/db/master.xml
I'll hand wave over these. Again, they should be cron'able, and probably
need to support various different output formats.
Removing documentation
If the user wants to remove a single document format, but keep the
information about the document, they might do
sk-remove-doc -format pdf -id MD5
which would remove the <docinstance> information for the PDF version of
the document with id == MD5.
If they want to remove a complete document, they do
sk-remove-doc -id MD5
which removes the whole thing. This does not touch the document's files
(such as /usr/local/share/doc/app-name/*), but it does update the Contents
List, and if the OMF has been stored in a separate file then it removes
that as well.
Normally, this behaviour would be carried out by pkg_delete, or whatever
packaging system the user is using.
Querying installed documentation
There should probably be a command line tool to allow the user to do
simple queries of the Contents List. Since the Contents List is in XML,
any XML aware tool will be able to handle it. However, sometimes you
just want to use awk.
sk-query -format csv
would dump the Contents List as CSV format.
id,name,format,path,...
Since the data is multi-dimensional, you would probably end up with
multiple lines with the same id and name, but different formats and paths.
It should be possible to query individual document IDs to retrieve
information about them;
sk-query -id MD5
and probably, given a pathname to a document, retrieve information about
it as well,
sk-query -path /usr/local/share/doc/app-name/html/index.html
Sanity checking the Contents List
There should be a tool to sanity check the Contents List, and warn the
end user of potential problems.
Several that spring immediately to mind.
1. Where a <docinstance> element exists, but for which no corresponding
files are found on the file system.
2. Files for which a <docinstance> element exists which points to
a remote document that can no longer be found.
3. I glossed over it earlier, but each <docinstance> should have a
modtime attribute, that contains the file's modification time.
Part of the sanity check should ensure that the modtime attribute
and the files (or the remote files) modification times do match.
If they don't, it's likely that the OMF data and/or the TOC and Index
are out of date, and need to be regenerated.
None of these are fatal errors, but they will cause the generated TOC and
Index to potentially be wrong. The tools should have options to fix the
errors themselves (in the case of (1) and (2)).
I hope all that makes sense. It's not too different from what's gone before,
but I hope the extra verbiage makes things a little clearer. In particular,
it (hopefully) dispenses with the path problem.
Before anyone asks, I'm not tied to any of the command names or the option
names that I've used here, nor am I particularly tied to the XML examples;
as long as we capture the content properly I don't care what the element
names are :-)
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|