Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Dan M. <d-...@uc...> - 2000-11-20 21:53:04
|
Nik,
WOW!
This is impressive. I read about 60% of it and then decided to print it
out for a closer look and so I can keep notes. So far it looks really
good.
Dan
On Mon, 20 Nov 2000, Nik Clayton wrote:
> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
>
> There should be no such thing as a non-relocatable package.
>
> By all means, allow default paths to be included, but they must always
> be overridable at installation time by the admin.
>
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
>
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
>
> 2. Maintaining a TOC from the Contents List
>
> 3. Maintaining an Index from the Contents List
>
> 4. Maintaining a Categories list
>
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
>
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
>
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
>
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
>
> 2. Locally installed system documentation, installed by the SA team.
>
> 3. Locally installed documentation, installed by the user.
>
> 4. Remote documentation on a web site.
>
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
>
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
>
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
>
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
>
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
>
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
>
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
>
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
>
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
>
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
>
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
>
> The OMF information may not even be on the same host as the target
> document.
>
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
>
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
>
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
>
> sk should treat this as one document with three formats, rather than
> three separate documents.
>
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
>
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
>
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
>
> -db The Contents List to update. Uses a default if not specified.
>
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
>
> This command does not physically copy the file in to place.
>
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
>
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
>
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
>
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
>
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
>
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
>
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
>
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
>
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
>
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
>
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
>
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
>
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
>
> Normally, this behaviour would be carried out by pkg_delete, or whatever
> packaging system the user is using.
>
> Querying installed documentation
>
> There should probably be a command line tool to allow the user to do
> simple queries of the Contents List. Since the Contents List is in XML,
> any XML aware tool will be able to handle it. However, sometimes you
> just want to use awk.
>
> sk-query -format csv
>
> would dump the Contents List as CSV format.
>
> id,name,format,path,...
>
> Since the data is multi-dimensional, you would probably end up with
> multiple lines with the same id and name, but different formats and paths.
>
> It should be possible to query individual document IDs to retrieve
> information about them;
>
> sk-query -id MD5
>
> and probably, given a pathname to a document, retrieve information about
> it as well,
>
> sk-query -path /usr/local/share/doc/app-name/html/index.html
>
> Sanity checking the Contents List
>
> There should be a tool to sanity check the Contents List, and warn the
> end user of potential problems.
>
> Several that spring immediately to mind.
>
> 1. Where a <docinstance> element exists, but for which no corresponding
> files are found on the file system.
>
> 2. Files for which a <docinstance> element exists which points to
> a remote document that can no longer be found.
>
> 3. I glossed over it earlier, but each <docinstance> should have a
> modtime attribute, that contains the file's modification time.
> Part of the sanity check should ensure that the modtime attribute
> and the files (or the remote files) modification times do match.
> If they don't, it's likely that the OMF data and/or the TOC and Index
> are out of date, and need to be regenerated.
>
> None of these are fatal errors, but they will cause the generated TOC and
> Index to potentially be wrong. The tools should have options to fix the
> errors themselves (in the case of (1) and (2)).
>
> I hope all that makes sense. It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer. In particular,
> it (hopefully) dispenses with the path problem.
>
> Before anyone asks, I'm not tied to any of the command names or the option
> names that I've used here, nor am I particularly tied to the XML examples;
> as long as we capture the content properly I don't care what the element
> names are :-)
>
> N
|
