Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Nik C. <ni...@no...> - 2000-11-23 00:59:35
|
On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > There should be no such thing as a non-relocatable package.
>
> Both relocatable and non-relocatable packages exist, so scrollkeeper
> should support them both in the simplest possible way.
Can you give me an example of non-relocatable package, and it's
justification for being non-relocatable?
> The current way things are designed is that it will have one contents list
> for the system.
I think that's a bad idea.
Also, I should be clear here -- I'm not talking about how the system is
currently designed, or what the code currently does. I'm describing
what I think SK should do, and what the interfaces should be. It's kind
of hard to evaluate the existing functionality at the moment, because I
haven't seen a roadmap that lists explicit design decisions, and things
that will and won't be supported.
> Of course you can always create another one somewhere
> else using the -p flag, but generally they would contain the same
> information and only the default one will get updated each time a document
> is installed. By specifying which OMF directories you want to include,
> these databases could be made to differ. We don't currently have a
> mechanism for supporting or specifying multiple OMF directories, but that
> would be fairly straightforward and is planned. This will make it very
> simple for users to have their own local databases which vary from the
> system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> which add to or over-ride the system defaults.
This now sounds different from design doc #3, which, IIRC, suggested
that the contents list, and the OMF information, would be kept separate.
> It is not clear how and why one would want to use multiple contents lists
> except for individual users and/or hand customization. The problem is
> that somebody has to specify what each database contains. So it works for
> users, because they can hand-tweak the contents. It doesn't work very
> well for system-wide databases, since everything must be specified by the
> packager. So while you could imagine GNOME documents being kept in one
> list and KDE documents being kept in another list, this doesn't really
> achieve anything.
I think it does, as long as a mechanism is provided so that sk can work
out where to put the information if the user doesn't indicate otherwise.
A config file that specifies the default Contents List should suffice.
> The information which would be passed on the command
> line to sk to indicate which list the doc goes in is really metadata which
> belongs in the OMF. OMF and XML provide everything we need to have one
> database and understand the scope of each piece. (eg. whether it is a
> GNOME doc or KDE doc.)
If you support multiple Contents Lists (and you have to, just to support
end users installing docs and not having write access to the system wide
Contents List) then you need to be able to specify which Contents List
to use.
Yes, you could put that information in the OMF, but, as I understand it,
the OMF contains meta information about the document. The location of
the Contents List is not meta information about the document, so should
not be in the OMF.
Also, consider a regular user who occasionally installs applications and
documentation in to $HOME.
It's much easier for them to add a flag to a command line (or, better
still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
than it is for them to edit OMF files before installing them.
> What is the advantage of breaking the main database into pieces?
Flexibility for the future.
> > 2. Maintaining a TOC from the Contents List
> >
> > 3. Maintaining an Index from the Contents List
> >
> > 4. Maintaining a Categories list
>
> Well, this is somewhat debatable and worthy of discussion. Somebody has
> to provide and maintain the categories list. On one side, we can just
> define it as part of OMF/sk and install a static list with sk. Then no
> maintenance is necessary. Or, we may consider having this specified by a
> configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> sys admin could drop in a replacement if they want. For example you could
> chop out all the HOWTO's and man pages if you know newbies who never touch
> a command line will be using your machine.
I don't think SK should mandate the existence of a categories list, or
mandate that certain categories should exist. I expect that some well
defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
"Gnome") will spring up.
However, that's not the same thing as saying
If a categories file exists (in some, as yet undecided format), SK
should have the option of warning the SA that they're about to install
a document with a category that does not currently exist, and asking
whether they want to continue. Naturally, the SA should have the
option of overriding this check with a command line flag.
> > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > application will have the responsibility for handling user queries of the
> > SK TOC and Index files ]
>
> sk should export searching functionality.
How? libsk.so? By having a program that accepts search parameters and
returns XML with the results? Some other mechanism?
> > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > maintain several different TOCs and Indices from several different Contents
> > Lists on the same host (for example, a regular user might want to run sk
> > to maintain their own index and TOC of ~/my-docs).
>
> For a given database, I agree. Eventually, I see the system having one
> database (with one contents list) and possibly each user having one. Is
> this what you have in mind? If you think we want more than one contents
> list in the system's database, please explain why.
Flexibility and efficiency. Suppose you have one big contents list that
contains information about all the documentation that's installed on the
system -- man pages, readmes, faqs, and so on. That's going to be a
very big contents list, particularly when apps like Gnome and KDE get in
on the act.
For example, on a BSD system, the contents of /usr/share/man change once
in a blue moon, as third party manual pages generally live under
/usr/local/share/man. So the admin might want to create a Contents List
for /usr/share/man once, and keep it separate, because the SA knows its
never going to change, so it doesn't need to be included in any
processes SK undertakes to ensure that the Contents List is sync with
the filesystem.
> > What sort of documentation can the Contents List going to point to?
> >
> > 1. Locally installed standard documentation that comes with the system,
> > such as man pages, or Info documents.
>
> Perhaps. I think we probably do want to include this information, using
> automagically generated (and therefore incomplete) OMF metadata as you
> suggest below. For man and info, we would have to update this information
> off of cron unfortunately. So there will be an unavoidable delay between
> when a document is installed and when it is registered with sk.
I don't quite agree. As I say below, I think installing a new document
should be a two step process, at least conceptually (as it could be
carried out by one command).
First, the user tells SK about the documentation.
Second, the user assigns some OMF to the documentation.
This is what I'm getting at with the sk-install-doc and sk-install-omf
command separation.
Consider the man page, for example. If I want to tell SK about all my
section 1 man pages, I'd like to be able to do something like
cd /usr/share/man/man1
sk-install-doc *.1
which would tell SK about all my section 1 manual pages. Actually, what
I'd probably do is something like
sk-install-doc -category "Man Pages:Section 1" *.1
to assign a category to them. At this point, there's no OMF for them,
but SK can synthesise the bare bones from the filename and the category.
Suppose, then, that I want to add some OMF for ls.1. I might do
something like this;
sk-extract-omf ls.1 > ls.omf
vi ls.omf
[edit edit edit]
sk-install-omf ls.1 ls.omf
which would pull out the OMF that SK synthesised for me in the earlier
step (presumably as XML). I can then edit it, and then install the
edited OMF back in to SK's control.
I think it's highly likely that *roff -man (and in particular, *roff
-mdoc) is going to be easy to extract OMF from. So what we might end up
with is a command to extract OMF from -man formatted documents. It
might work like this;
cd /usr/share/man/man1
foreach man in (*.1)
extract-omf -source man $man > `basename $man .1`.omf
end
which would iterate over *.1, extracting the OMF and writing it to
*.omf. With that done, the end user might then be able to do
foreach man in (*.1)
sk-install-doc -doc $man -omf `basename $man .1`.omf
end
which could install the document, and the pregenerated OMF, into SK with
one command.
I hope the design ethos that's coming across from all of this is to have
lots of small tools, each of which does one thing.
> > 2. Locally installed system documentation, installed by the SA team.
>
> SA =? sys admin
Yes.
> > 3. Locally installed documentation, installed by the user.
>
> Yes. Although like much of this discussion, this feature will come in a
> post-0.1 release.
As I say, I'm not thinking in terms of 'release' yet. I'd like to get
some specific functionality nailed down and agreed on.
[ Incidentally, and talking about the implementation -- one of the
things that occured to me last night; sk could have a command
structure like that of cvs, where instead of having a bunch of
sk-foo-bar commands, you have one 'sk' command, that takes options,
and then subcommands, which also take options.
sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
sk install-omf /path/to/omf
and so on. ]
> > I think it is fair to say that, when run, sk will need to construct it's
> > global Contents List by parsing multiple, smaller, contents lists. In much
> > the same way that SGML catalog files can include one another at the moment.
>
> Why? It sounds very nice and wonderful. However, this adds a layer of
> complexity without any obvious benefits.
As above.
Note that I don't mean that SK goes and writes a new, master contents
list file. Just that, in memory, SK is capable of aggregating
information from multiple Contents Lists (which should be accessible by
URL, so that remote Contents Lists are feasible).
> > DD: sk must have a run time option to allow the end user to select which
> > TOC to update.
>
> The databases are stored together. So, this should be specified at the
> same time as the contents list part of the database. I don't think I want
> my contents list coming from one database and my toc from another.
>
> (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> TOC.)
It's my understanding that there's a 1:1 mapping between a Contents List
and a ToC. That is, you can't have a ToC that includes information
about documents that aren't in the Contents List.
So if you have multiple Contents Lists, you need to have multiple ToCs
(and Indexes), and a way of saying that >this< Contents List was used to
produce >This< ToC.
Or are you saying that the ToC should be stored within the Contents List?
> > DD: sk must have a run time option to allow the end user to select which
> > Index to update.
>
> (as above)
Ditto :-)
> > DD: One of the sk commands must be to generate a usable Contents List
> > (which really means {X}HTML version, to start with) given a Contents
> > List which is in sk internal format.
>
> Everything is XML right now, so this is not an issue. If we decide
> another internal format is better we should still export XML to the
> browser. As for HTML, this is possible and may be a nice interface for
> people who want to create a very minimalistic web browser.
All the world is not yet XML, and not everyone runs KDE or Gnome. Being
able to convert the ToCs and Indices to other formats is vital.
Fortunately, since the ToC and Indices are XML, this should be easy,
using something like XSLT, so we can probably just consider this
functionality that we get for free.
> > DD: It is possible for a document to change, but for the OMF information
> > to be neglected. sk should maintain a modification time stamp for
> > each document and its corresponding OMF, to allow the end user to
> > determine when that might have happened, and to correct for it.
>
> For the current version of sk, which does not yet allow for remote
> documents, only the OMF file's timestamp is watched. This is because if
> somebody edits the doc and not the OMF, then it is their fault for not
> updating the OMF, not sk's. In reality whenever you install a new version
> of a package, even if the OMF is not updated (as it should), it still
> updates the old version so it gets a new timestamp and the new doc gets
> registered despite any negligence on the part of the maintainer to update
> the OMF (or the doc for that matter;).
>
> This mainly becomes an issue for:
> 1) remote docs
> 2) docs which users write and modify themselves.
>
> So we may indeed need to start tracking the doc timestamps when we support
> these features. We have to be careful about constantly checking the
> timestamps on a bunch of docs which are spread around the internet
> though. There will be many cases where this task cannot complete, or else
> takes an unacceptable amount of time.
This should be configurable behaviour, something like;
sk --check-remote-timestamps --remote-timeout=30
Actually, this implies another DD
DD: sk must track the validity of the OMF in relation to it's document.
OMF might be:
valid This OMF was installed with the document, and neither
the document or the OMF's timestamps have changed.
invalid The document and OMF have different timestamps
unknown The document and OMF have the same timestamps, but
the most recent attempt to confirm this (particularly
for remote documents) failed.
sk should be able to report on documents who have OMF which is not
valid, so the SA can investigate further.
> > DD: One document, multiple output formats. The user might have installed
> > the same document three times, once as HTML, once as PDF, and once as
> > plain text. The three installations are the same document, and have
> > the same OMF associated with them.
>
> No. These definitely do not have the same OMF, unless they are identical
> copies. The OMF file specifies the document format in FORMAT.DTD and
> FORMAT.MIME. The docs also have different IDENTIFIER's.
>
> (We may decide to expand this a little bit. FORMAT.STYLESHEET may be
> nice, as it would allow the doc authors to dictate the appearance of their
> doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> icon, etc.)
>
> > sk should treat this as one document with three formats, rather than
> > three separate documents.
>
> If you look in the ldp-meta archives from several months ago, I proposed
> we add a document id to the OMF. This would allow us to identify if two
> documents are "The-Same". I call two documents "The-Same" if they are the
> same document only different versions, languages, formats, etc. If two
> documents have the same title, they are not necessarily
> "The-Same". Likewise, if I change the title of my document at some point
> in time, I can have two docs which are "The-Same" but not having the same
> title.
>
> Obviously this sort of information will be very nice to have. You can do
> things like "Get me this document in language xxx." or "Show me the latest
> version of this document on the Net." or "Show me the previous version of
> this document from the Net."
>
> > [ This is probably going to be the most contentious piece of this
> > writeup, and it might change. In particular, it also suggests that
> > translations of a document should be treated as the same document,
> > just in a different language, and I'm not sure that's a good idea.
> > Then again, it might be. ]
>
> I guess we were thinking along similar lines :)
Do you agree that making the distinction between a document, and one or
more instances of that document (different formats, different languages)
is worthwhile.
> > There are a number of tasks that the sk end user is going to want to tell
> > sk to do. They are;
> >
> > Install new documentation that comes bundled with OMF
> >
> > The user has just downloaded and installed a package that comes with one
> > or more bundled documents, and the package maintainer has thoughtfully
> > provided an OMF file for each document.
> >
> > Each document has been provided in HTML and PDF format.
> >
> > Suppose the user installs the HTML documentation first, and installs it
> > it into /usr/local/share/doc/app-name/html/, where normal file is called
> > index.html.
> >
> > [ Everywhere I say "the user runs" I really mean "The package maintainer
> > does this by means of a post-install script that they've already written.
> > It is assumed that this post-install script knows the directories that
> > documentation has been installed in, by virtue of being part of the
> > package that did the installation in the first place. ]
> >
> > The user runs
> >
> > sk-install-doc -format html-split -db /var/db/sk/master.xml \
> > /usr/local/share/doc/app-name/html/index.html
>
> Why do we specify the format on the command line? The FORMAT is specified
> in the OMF already. The packager should just install and register the
> metadata instead of putting it on the command line.
Because at this point, all we've done is install a document with no OMF.
SK has to synthesise some OMF, based on any additional information we
give it.
This separation is necessary to support documents that have no separate
OMF. Adding "-format" to the command line is probably going to be
easier for an end user than having to create a separate file to return
to.
> > 'sk-install-doc' registers the document. The two (optional) arguments
> > here are
> >
> > -format The format of the installed documentation. Might be
> > html-split - Bunch of small HTML files
> > html - One big HTML file
> > ps - postscript
> > pdf - PDF
> > ... - extend as necessary (rtf, txt, pdb, ...)
> >
> > If not specified, sk can try and guess what it is.
>
> I don't think we really want this. One could make an argument that if no
> OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
> has a bunch of arguments for each element in the OMF. However I'm not
> sure if we want to merge all this into the same command that registers the
> document with sk.
Six of one, half a dozen of another. Certainly, a sequence that went
something like
sk-build-omf -format html ... > doc.omf
sk-install-doc <options here> index.html
sk-install-omf <options here> doc.omf
would also work.
> > -db The Contents List to update. Uses a default if not specified.
>
> Here I assume (hope?;) that you mean the database directory and not the
> contents list. If so, we already do this with the "-p" flag.
Right.
> > [ Note: This also needs a category option, so that the end user can
> > specify which categories the document goes in to. By default,
> > it goes in to "unfiled". ]
>
> Here we go, implementing the OMF on the command-line ;)
> We should put all this metadata in the OMF metadata file.
I'd like to see a mechanism to specify it on the command line (apart
from anything else, it's easier to script). sk-build-omf, as you
suggest above, seems like a good compromise.
> > This command does not physically copy the file in to place.
> >
> > sk-install-doc should generate a unique identifier for this document
> > (MD5 hash of the filename?), and display this to the user. We probably
> > also need a standalone utility (sk-generate-id?) that, given a path,
> > generates an ID.
>
> Right now we just number them, which guarantees a unique identifier.
Two problems with this:
1. Probably won't scale efficiently as the Contents List gets larger.
2. Makes it impossible to merge two separate Contents Lists, from two
or more sites (say you're on a univesity campus, where the biology
department and the chemistry department both make available their
list of documentation in two different Scroll Keeper Contents Lists).
Your document browser is going to need a mechanism to uniquely
identify each document. If there's a strong probability that the
document IDs are not unique then the browsers are each going to
have to reinvent their own mechanism for ensuring that the
documents are unique.
The documents URL might be appropriate, but it's a bit unwieldy.
You really need a mechanism that can (as near as possible) guarantee
globally unique document IDs.
> > As you can see, not a lot of information in there so far. "lang" and
> > "encoding" could also be specified on the command line, and if not
> > specified, some defaults are used.
> >
> > I think we probably also need a modification time attribute, but I haven't
> > shown that here.
>
> The modification time is stored in a seperate file which keeps track of
> which documents are in the database and is used for database maintenance,
> updating, installing, uninstalling, etc. The contents list does not care
> about the modification time.
It probably should. Apart from the issues I outline above, this then
also lets you do things like "Show me the documents in order, most
recently modified first".
> Hopefully we agree that sk should be as simple and robust for the packager
> and distributor as possible. This means the least amount of typing with
> the smallest possible learning curve, with good fault tolerance.
Yes. Having said that, we shouldn't be afraid to aggregate certain
functionality in to one tool.
For example, cp(1), mv(1), chmod(1), and chown(1) all do one task, and
do it well. That doesn't stop install(1) from being useful in its own
right.
> > OK, so we've told sk about the document, but we haven't pointed to
> > any of the OMF data yet. Recall that the OMF data is the same despite
> > the different document formats.
>
> Actually they are different OMF files which have different IDENTIFIER's
> and FORMAT's.
OK. I'm going to poke around OMF in more detail, and think about this
issue some more.
> > ==========================================================================
> > Category: Unfiled
> >
> > The App-Name manual
> >
> > Include meta information from the OMF here (SK ID = MD5)
> >
> > Formats: Local HTML (split), PDF, Remote HTML (split)
> > =========================================================================
>
> This may be useful for people who don't have a help-browser aside from
> Netscape. Although Netscape won't know how to display most docs.
True. However, we should allow the Index builder flexibilit in deciding
what documents are included in the Index. This may include producing a
special Index that only contains documents available in HTML, plain
text, and PDF (for example).
> > Generate TOC and Index
> >
> > sk-generate-toc -format html -db /var/db/master.xml
> >
> > sk-generate-index -format html -db /var/db/master.xml
>
> As with the contents list, we could export html. However I think most
> browsers will find XML much more useful.
As I say, this is a simple XSLT transformation. As long as the source
format is XML this should be relatively simple.
> I'm unclear on how and why we should embed contents lists inside each
> other.
Hopefully I've answered that.
> I'm also very scared by the idea of listing metadata on the
> command line in addition to inside the OMF metadata file. I think this
> creates a lot more work for packagers and risks introducing a lot of
> errors and inconsistancies in the metadata without gaining anything. The
> obvious exception is the case where an OMF file does not exist, but I
> think creating the metadata and installing it should be two seperate
> steps.
I'm only considering it for the cases where there is no OMF, and you
need to bootstrap it in to SK.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|