Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Laszlo K. <las...@su...> - 2000-11-21 13:34:50
|
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
Yes, but we didn't aim for more than one.
> 2. Maintaining a TOC from the Contents List
Yes.
> 3. Maintaining an Index from the Contents List
Yes.
> 4. Maintaining a Categories list
What is the difference between this and (1)?
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
>
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
Multiple content lists would probably be a lower priority.
>
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
>
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
It will be able to categorize them if OMF files are created for them. I
think sk wants to categorize docs based on OMF files while the ones
without OMF files would not be considered by it. They can be considered
by the help browser though as it happens now with Nautilus in Gnome.
> 2. Locally installed system documentation, installed by the SA team.
>
> 3. Locally installed documentation, installed by the user.
>
> 4. Remote documentation on a web site.
Yes to all.
>
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
As I said before sk works with one Content List at moment and we did not
think about more than one. You might want to detail how you imagine
this.
>
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
>
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
>
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
Waht do you mean about TOC update? At the moment this is extracted at
install time (from sgml only) and stored in a file. As long as the doc
does not change, the TOC does not need to be updated.
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
Saem as for TOC above,
>
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
The sk Content List is XML, why would we want to generate HTML? I guess
it depends on the browser what kind of files are needed. Anyway I am
sure our XML Content List can be easily converted to HTML.
>
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
>
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
This has to be discussed. OMF is the heart of Scrollkeeper, I am not
sure we intend to make Scrollkeeper to deal with docs that don't have
OMFs. But as I said before, the browser can deal with them separately.
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
Yes, that is a good idea.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
>
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
This is planned for SGML, it can be done for other formats too.
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
Yes, this is how it works.
> The OMF information may not even be on the same host as the target
> document.
At the moment the OMF information is local, the document might not be.
However the current implementation supports only local docs.
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
Currently we take for granted that the OMF is changed at any doc change.
If the doc has an OMF then when the whole package is re-installed then
the OMF changes (the timestamp, this is what we monitor).
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
I think this is working. You can certainly install a doc in multiple
places in the Content List.
>
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
>
> sk should treat this as one document with three formats, rather than
> three separate documents.
>
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
It doesn't work like this right now, every doc needs a separate OMF
file. However one OMF file with several doc entries each describing a
different doc is supported. So it would be possible to use one OMF file
for the same doc in multiple formats.
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
>
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
>
> -db The Contents List to update. Uses a default if not specified.
>
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
>
> This command does not physically copy the file in to place.
I guess here you mean that most or all of the OMF metadata should be
able to be passed in as a parameter in the installation script/binary.
Good idea.
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
md5 was ruled out before because it is too slow on Solaris. But we have
a mechanism of assigning unique IDs by storing the assigned ones and
then just not reassigning them again.
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
>
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
The current implementation assigns different IDs for the same doc in
different formats. However I don't see this as a problem as when we
uninstall docs the OMF files are deleted, scrollkeeper then uninstall
all the related docs.
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
That sounds good.
>
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
>
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
>
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
The current process is much simpler than this and it achieves the same
thing. Maybe you should also look in more detail to the current
implementation.
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
>
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
>
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
>
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
All of this can be implemented, but I don't see it too important for the
moment.
>
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
>
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
I guess we can support this later on.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
>
> Normally, this behaviour would be carried out by pkg_delete, or whatever
> packaging system the user is using.
This is fully automatic, if you delete the OMF file, all the related
docs will be uninstalled.
>
> Querying installed documentation
>
> There should probably be a command line tool to allow the user to do
> simple queries of the Contents List. Since the Contents List is in XML,
> any XML aware tool will be able to handle it. However, sometimes you
> just want to use awk.
>
> sk-query -format csv
>
> would dump the Contents List as CSV format.
>
> id,name,format,path,...
>
> Since the data is multi-dimensional, you would probably end up with
> multiple lines with the same id and name, but different formats and paths.
>
> It should be possible to query individual document IDs to retrieve
> information about them;
>
> sk-query -id MD5
>
> and probably, given a pathname to a document, retrieve information about
> it as well,
>
> sk-query -path /usr/local/share/doc/app-name/html/index.html
Good idea.
>
> Sanity checking the Contents List
>
> There should be a tool to sanity check the Contents List, and warn the
> end user of potential problems.
>
> Several that spring immediately to mind.
>
> 1. Where a <docinstance> element exists, but for which no corresponding
> files are found on the file system.
>
> 2. Files for which a <docinstance> element exists which points to
> a remote document that can no longer be found.
>
> 3. I glossed over it earlier, but each <docinstance> should have a
> modtime attribute, that contains the file's modification time.
> Part of the sanity check should ensure that the modtime attribute
> and the files (or the remote files) modification times do match.
> If they don't, it's likely that the OMF data and/or the TOC and Index
> are out of date, and need to be regenerated.
>
> None of these are fatal errors, but they will cause the generated TOC and
> Index to potentially be wrong. The tools should have options to fix the
> errors themselves (in the case of (1) and (2)).
Good idea.
> I hope all that makes sense. It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer. In particular,
> it (hopefully) dispenses with the path problem.
Yes it helps a lot. You look like having lots of experience in the area.
Laszlo
|