Menu
▾
▴
scrollkeeper-devel
|
From: Dan M. <d-...@uc...> - 2000-11-17 16:46:28
|
Hi everybody,
I've been trying to think of a solution to the problem of our current
ScrollKeeper design not handling relocatable packages properly. (ie.
packages where the installation location can be determined at install
time, after the package was built). I think I have a solution.
Before I get into my idea, I wanted to ask people who actually supports
and uses relocatable packages? Apparently all of Red Hat Linux and GNOME
are non-relocatable. Nick has indicated that *BSD at least sometimes uses
relocatable packages. I'm curious which operating system / packaging
system / distributions actually support and use relocatable packages.
My suggestion:
For non-relocatable packages, everything stays exactly as it is now and
was described in proposal #3.
For relocatable packages, we provide an extra script
('scrollkeeper-update-identifier') which can be called as a post-install
script before 'scrollkeeper-update' to "fix" the doc path in the
IDENTIFIER field of the OMF file. This makes a little bit more work to
make your package relocatable, but I don't think that should surprise
anybody.
To make things as convenient as possible, we probably want to make
scrollkeeper-update-identifier flexible to handle a few different common
ways a person would want to use it:
1) scrollkeeper-update-identifier <omffile> <doc>
eg: scrollkeeper-update-identifier /usr/share/omf/foo.omf
/use/local/doc/foo/foo.sgml
It simply verifies that <doc> exists and then sticks <doc> into <omffile>
as the IDENTIFIER.
2) scrollkeeper-update-identifier <omffile> <path>
eg: scrollkeeper-update-identifier /usr/share/omf/foo.omf
/use/local/doc/foo/
It first reads the IDENTIFIER in <omffile>.
A) If the IDENTIFIER is just a file name it looks in <path> for the file.
If it does not find it there, it descends any subdirectories, searching
for the doc.
B) If the IDENTIFIER is a path with a file name, it tacks that onto <path>
and then looks for that file. This way a person could set up all their
OMF files initially to hold relative paths to the doc from the main
director(ies) they are installed in. So the relative paths are specified
at build time, and the toplevel directory for the doc is specified at
install time.
I think this should make it fairly easy for people to use. If you have
many docs, you would probably want to place relative paths in the original
OMF files. Then in the post-install part of the package, loop over all
these OMF files passing the <path> under which all of the docs are
installed. (Similar to what was done in the 'info' installation example
Nick pointed out.)
After all the OMF files have their IDENTIFIERs updated, one runs
'scrollkeeper-update' as usual.
Does this sound reasonable to everybody? Feedback is welcome :)
Dan
|
|
From: Nik C. <ni...@no...> - 2000-11-17 17:47:05
|
On Fri, Nov 17, 2000 at 10:46:20AM -0600, Dan Mueth wrote: > Before I get into my idea, I wanted to ask people who actually supports > and uses relocatable packages? Apparently all of Red Hat Linux and GNOME > are non-relocatable. Nick has indicated that *BSD at least sometimes uses > relocatable packages. I'm curious which operating system / packaging > system / distributions actually support and use relocatable packages. You *absolutely* have to support relocatable packages. The BSD ports/packages systems support it, as does RPM. I'd be incredibly surprised if Debian's format didn't. For example, suppose I want to install a Debian administrator's guide on to my FreeBSD system (perhaps I want to learn more about Debian). If I read the Debian docs properly, it's going to want to install somewhere under /usr/doc. Sorry, on a FreeBSD system that's a no go area -- /usr/doc is reserved for documentation that comes with the OS (in fact, it's not even /usr/doc anymore it's /usr/share/doc/, because /usr/share/ is where architecturally neutral files live). Since this is a local addition to the system, it would (on FreeBSD) live in /usr/local/share/doc/<mumble>. Of course, I might not be root on this box, in which case I'm probably going to want to install it somewhere under $HOME/share/doc/, or similar (or maybe even $HOME/public_html, so I can browse it easily from elsewhere). > My suggestion: > > For non-relocatable packages, everything stays exactly as it is now and > was described in proposal #3. There should be no such thing as a non-relocatable package. For FreeBSD, this means that some of the guys that produce the ports do actually go through the code, substituting "/usr/bin" with "$PREFIX/bin" (or whatever) so that the application can be installed outside of it's expected directory hierarchy. N -- Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95. Telephone line, $24.95 a month. Software, free. USENET transmission, hundreds if not thousands of dollars. Thinking before posting, priceless. Somethings in life you can't buy. For everything else, there's MasterCard. -- Graham Reed, in the Scary Devil Monastery |
|
From: Dan M. <d-...@uc...> - 2000-11-17 18:27:45
|
On Fri, 17 Nov 2000, Nik Clayton wrote: > On Fri, Nov 17, 2000 at 10:46:20AM -0600, Dan Mueth wrote: > > Before I get into my idea, I wanted to ask people who actually supports > > and uses relocatable packages? Apparently all of Red Hat Linux and GNOME > > are non-relocatable. Nick has indicated that *BSD at least sometimes uses > > relocatable packages. I'm curious which operating system / packaging > > system / distributions actually support and use relocatable packages. > > You *absolutely* have to support relocatable packages. My questions wasn't *whether* to support relocatable packages. (As you can see I proposed a method on how we can support them.) Instead I was asking who makes packages which are relocatable. Presumably whoever replies with "We use relocatable packages." will have a vested interest in helping us find the best way to support them. > The BSD ports/packages systems support it, as does RPM. I'd be incredibly > surprised if Debian's format didn't. > > For example, suppose I want to install a Debian administrator's guide > on to my FreeBSD system (perhaps I want to learn more about Debian). > > If I read the Debian docs properly, it's going to want to install somewhere > under /usr/doc. > > Sorry, on a FreeBSD system that's a no go area -- /usr/doc is reserved for > documentation that comes with the OS (in fact, it's not even /usr/doc anymore > it's /usr/share/doc/, because /usr/share/ is where architecturally neutral > files live). > > Since this is a local addition to the system, it would (on FreeBSD) live > in /usr/local/share/doc/<mumble>. Of course, I might not be root on this > box, in which case I'm probably going to want to install it somewhere > under $HOME/share/doc/, or similar (or maybe even $HOME/public_html, so > I can browse it easily from elsewhere). > > > My suggestion: > > > > For non-relocatable packages, everything stays exactly as it is now and > > was described in proposal #3. > > There should be no such thing as a non-relocatable package. Perhaps. However that is more idealogical. In reality many (possibly most?) packages are non-relocatable. For example all of Red Hat Linux is non-relocatable. (I have no idea about the other Linux distributions or Solaris.) Those people who already have non-relocatable packages will leave out the step which makes their docs relocatable. > For FreeBSD, this means that some of the guys that produce the ports do > actually go through the code, substituting "/usr/bin" with "$PREFIX/bin" > (or whatever) so that the application can be installed outside of it's > expected directory hierarchy. So what do you think of my suggestion for how we allow people to make relocatable packages which use scrollkeeper? Does it seem like a decent solution? Do you have any ideas on a better solution or how we can improve this one? It seems to me that it makes things substantially easier than the original suggestion which required the install script to keep track of the location of every OMF file and every document. At the same time it keeps things as simple as possible for those people who do not make relocatable packages. Dan |
|
From: Laszlo K. <las...@su...> - 2000-11-20 14:48:30
|
[snip] > There should be no such thing as a non-relocatable package. I guess what we are trying to do is to create support for both relocatable and non-relocatable packages. It is up to the developer and the packager to decide if and how they want to use this support. Laszlo |
|
From: Nik C. <ni...@no...> - 2000-11-20 21:21:43
|
On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> [snip]
> > There should be no such thing as a non-relocatable package.
>
> I guess what we are trying to do is to create support for both
> relocatable and non-relocatable packages. It is up to the developer and
> the packager to decide if and how they want to use this support.
There should be no such thing as a non-relocatable package.
By all means, allow default paths to be included, but they must always
be overridable at installation time by the admin.
Imagine an RPM that wanted to install in /opt, and wouldn't let you install
it anywhere else.
Based on the design docs on the web, what does sk have responsibility for:
[ This is my current understanding. Please correct any misconceptions I
might have. Things I flag with 'DD' are design decisions that I think
fall out of the model. ]
1. Maintaining one or more lists of documentation that the user wants
a TOC for (the "Contents List" from the most recent proposal)
2. Maintaining a TOC from the Contents List
3. Maintaining an Index from the Contents List
4. Maintaining a Categories list
[ Note: I've explicitly ommitted searching from this list -- AIUI, someother
application will have the responsibility for handling user queries of the
SK TOC and Index files ]
Activities (2) and (3) require a Contents List. Note that a don't say "the
{TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
maintain several different TOCs and Indices from several different Contents
Lists on the same host (for example, a regular user might want to run sk
to maintain their own index and TOC of ~/my-docs).
What sort of documentation can the Contents List going to point to?
1. Locally installed standard documentation that comes with the system,
such as man pages, or Info documents.
QUESTION: Is sk going to deal with non-SGML/XML documents, such as
these?
2. Locally installed system documentation, installed by the SA team.
3. Locally installed documentation, installed by the user.
4. Remote documentation on a web site.
I think it is fair to say that, when run, sk will need to construct it's
global Contents List by parsing multiple, smaller, contents lists. In much
the same way that SGML catalog files can include one another at the moment.
DD: Contents Lists must be able to refer to other Contents Lists, and sk
must have an option to chase the link to read the referred to Contents
List.
DD: sk must have a run time option to allow the end user to select which
Contents Lists the user wants sk to process.
DD: sk must have a run time option to allow the end user to select which
TOC to update.
DD: sk must have a run time option to allow the end user to select which
Index to update.
DD: One of the sk commands must be to generate a usable Contents List
(which really means {X}HTML version, to start with) given a Contents
List which is in sk internal format.
Each piece of documentation (which might really consist of tens or even
hundreds of files) can be considered to consist of two parts.
1. The documentation itself (*.html, *.png, foo.pdf, whatever).
2. The OMF, containing the 17 (?) key pieces of meta information that
sk likes to keep about the documentation. Not all 17 pieces of
information may necessarily be present.
Not all documentation the user wants to point to is going to have the OMF
information associated with it. This is unavoidable.
DD: sk must work tolerably well with minimum OMF information. At the
worst case, sk must do something useful when the only OMF information
available is the document title, and a URL pointing to the document's
'index' page.
Actually, I take that back. The only piece of information sk needs
is the URL. In the worst case (with 0 OMF information) the URL
becomes the title, until the end user provides some OMF, or points
sk at pre-existing OMF.
DD: sk should probably include some support applications which can parse
common documentation formats (*roff man and mdoc, GNU info, HTML,
LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
them as possible.
In some cases this is necessary to produce bare bones OMF files that
will not be processed further. In other cases, this will be to produce
OMF files that will be further customised and improved by the end user.
DD: sk should include some support applications that can generate TOC and
Index information from a variety of source formats.
DD: sk should not assume that the OMF information is in the same directory
as the document, nor should it assume that the location of the OMF
information can be inferred based on the document's location. The
location of the OMF information must always be explicitly listed in
the Contents List.
The OMF information may not even be on the same host as the target
document.
DD: It is possible for a document to change, but for the OMF information
to be neglected. sk should maintain a modification time stamp for
each document and its corresponding OMF, to allow the end user to
determine when that might have happened, and to correct for it.
DD: An application may have multiple pieces of documentation associated
with it. Applications are also not the only things that have
documentation, the system does (think: man page sections). Simply
storing information about a document isn't enough, you need to be
able to group documents into chunks (and probably group chunks into
chunks, or allow the same document to appear in multiple categories).
I'm not going to discuss this in too much detail in this document, this
is the reason for "4. Maintaining a Categories list" earlier.
DD: One document, multiple output formats. The user might have installed
the same document three times, once as HTML, once as PDF, and once as
plain text. The three installations are the same document, and have
the same OMF associated with them.
sk should treat this as one document with three formats, rather than
three separate documents.
[ This is probably going to be the most contentious piece of this
writeup, and it might change. In particular, it also suggests that
translations of a document should be treated as the same document,
just in a different language, and I'm not sure that's a good idea.
Then again, it might be. ]
SK Tasks
--------
There are a number of tasks that the sk end user is going to want to tell
sk to do. They are;
Install new documentation that comes bundled with OMF
The user has just downloaded and installed a package that comes with one
or more bundled documents, and the package maintainer has thoughtfully
provided an OMF file for each document.
Each document has been provided in HTML and PDF format.
Suppose the user installs the HTML documentation first, and installs it
it into /usr/local/share/doc/app-name/html/, where normal file is called
index.html.
[ Everywhere I say "the user runs" I really mean "The package maintainer
does this by means of a post-install script that they've already written.
It is assumed that this post-install script knows the directories that
documentation has been installed in, by virtue of being part of the
package that did the installation in the first place. ]
The user runs
sk-install-doc -format html-split -db /var/db/sk/master.xml \
/usr/local/share/doc/app-name/html/index.html
'sk-install-doc' registers the document. The two (optional) arguments
here are
-format The format of the installed documentation. Might be
html-split - Bunch of small HTML files
html - One big HTML file
ps - postscript
pdf - PDF
... - extend as necessary (rtf, txt, pdb, ...)
If not specified, sk can try and guess what it is.
-db The Contents List to update. Uses a default if not specified.
[ Note: This also needs a category option, so that the end user can
specify which categories the document goes in to. By default,
it goes in to "unfiled". ]
This command does not physically copy the file in to place.
sk-install-doc should generate a unique identifier for this document
(MD5 hash of the filename?), and display this to the user. We probably
also need a standalone utility (sk-generate-id?) that, given a path,
generates an ID.
At this point, /var/db/sk/master.xml looks something like this:
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
</doc>
As you can see, not a lot of information in there so far. "lang" and
"encoding" could also be specified on the command line, and if not
specified, some defaults are used.
I think we probably also need a modification time attribute, but I haven't
shown that here.
Now, the user uses sk-install-doc again, but this time they install the
PDF file.
sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
/usr/local/share/doc/app-name/pdf/doc.pdf
A very similar command line. However, because this is another instance of
the same document, this time the user specifies the ID of the document.
master.xml now looks like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
format="pdf" lang="en_US" encoding="ISO_8859-1">
</doc>
Finally, the chap that wrote the application also maintains information
about the document on the web. So the user decides to do the following;
sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
http://www.example.com/app-name/index.html
and master.xml now looks like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
format="pdf" lang="en_US" encoding="ISO_8859-1">
<docinstance url="http://www.example.com/app-name/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
</doc>
Now, typically, the end user wouldn't run any of this themselves. It
is the responsibility of whoever produced the package/rpm/whatever to
make sure that this happens.
I understand that some people have said that that's too much work to
expect package maintainers to do. I don't think that's the case. It
doesn't matter if the original program's author, or the package maintainer
doesn't want to do it. All it takes is someone to find this functionality
useful enough to spend the five minutes constructing the sk-* command
lines to do this, and submit it back to the maintainer. These sorts of
commands are not going to change a great deal between package releases.
[ In fact, I see this as being a differentiator between operating systems.
SuSE (to pick an example) might not bother generating OMF files for
applications that don't already provide them. But it might be a point
of pride for the Debian people that they don't bundle a third party
application until the person making the .deb file has generated an OMF
file for it ]
OK, so we've told sk about the document, but we haven't pointed to
any of the OMF data yet. Recall that the OMF data is the same despite
the different document formats.
The end user (or their package management application) runs
sk-install-omf -id MD5 app-name.omf
Notice how the MD5 checksum (or whatever) is included on the command line,
so that sk can associate this OMF file with the document that was installed
earlier.
Now, because we've already installed the document, and sk knows where the
document has been installed in to, the OMF shouldn't need to contain any
path data. So there's nothing for sk to need to rewrite. sk can either
copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
it master.xml, or it can copy it wholesale in to master.xml.
The former might look like this;
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<!-- Other <docinstance> elements here -->
<omf url="file:/var/db/sk/omf/<md5>.omf">
</doc>
the latter might look like
<doc docid="the MD5 checksum">
<docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
format="html-split" lang="en_US" encoding="ISO_8859-1">
<!-- Other <docinstance> elements here -->
<omf>
<omf:title>The App-name manual</omf:title>
<!-- Other elements here -->
</omf>
</doc>
It shouldn't matter to the end user.
The user can now regenerate their HTML list of documents installed on
the system. They might do something like
sk-generate-contents -format html -db /var/db/master.xml > contents.html
which would generate contents.html, which, in a browser, might look
something like this;
==========================================================================
Category: Unfiled
The App-Name manual
Include meta information from the OMF here (SK ID = MD5)
Formats: Local HTML (split), PDF, Remote HTML (split)
=========================================================================
I anticipate that there will be a system wide sk Contents List (like
/var/db/sk/master.xml in my examples) that will be used to generate
a system wide HTML (and other formats) Contents List regularly, through
cron, or something similar.
Notice that all this works whether the end user is root or not. If the
documentation has been installed somewhere under $HOME then the user
can adjust sk-install paths as necessary.
Install new documentation that doesn't come with OMF
This is very similar to the previous example. However, instead of
running sk-install-omf, the user has two choices.
1. Skip the step entirely. In which case, when the contents list is
generated there won't be a document title, just a filename.
2. Run one of the helper apps I talked about earlier, that tries to
parse the document and make a stab at generating the OMF. The
user can then clean this up hand as necessary, and run sk-install-omf
by hand (if they're so inclined). They can also submit the OMF
they've written back to the original author.
Generate TOC and Index
sk-generate-toc -format html -db /var/db/master.xml
sk-generate-index -format html -db /var/db/master.xml
I'll hand wave over these. Again, they should be cron'able, and probably
need to support various different output formats.
Removing documentation
If the user wants to remove a single document format, but keep the
information about the document, they might do
sk-remove-doc -format pdf -id MD5
which would remove the <docinstance> information for the PDF version of
the document with id == MD5.
If they want to remove a complete document, they do
sk-remove-doc -id MD5
which removes the whole thing. This does not touch the document's files
(such as /usr/local/share/doc/app-name/*), but it does update the Contents
List, and if the OMF has been stored in a separate file then it removes
that as well.
Normally, this behaviour would be carried out by pkg_delete, or whatever
packaging system the user is using.
Querying installed documentation
There should probably be a command line tool to allow the user to do
simple queries of the Contents List. Since the Contents List is in XML,
any XML aware tool will be able to handle it. However, sometimes you
just want to use awk.
sk-query -format csv
would dump the Contents List as CSV format.
id,name,format,path,...
Since the data is multi-dimensional, you would probably end up with
multiple lines with the same id and name, but different formats and paths.
It should be possible to query individual document IDs to retrieve
information about them;
sk-query -id MD5
and probably, given a pathname to a document, retrieve information about
it as well,
sk-query -path /usr/local/share/doc/app-name/html/index.html
Sanity checking the Contents List
There should be a tool to sanity check the Contents List, and warn the
end user of potential problems.
Several that spring immediately to mind.
1. Where a <docinstance> element exists, but for which no corresponding
files are found on the file system.
2. Files for which a <docinstance> element exists which points to
a remote document that can no longer be found.
3. I glossed over it earlier, but each <docinstance> should have a
modtime attribute, that contains the file's modification time.
Part of the sanity check should ensure that the modtime attribute
and the files (or the remote files) modification times do match.
If they don't, it's likely that the OMF data and/or the TOC and Index
are out of date, and need to be regenerated.
None of these are fatal errors, but they will cause the generated TOC and
Index to potentially be wrong. The tools should have options to fix the
errors themselves (in the case of (1) and (2)).
I hope all that makes sense. It's not too different from what's gone before,
but I hope the extra verbiage makes things a little clearer. In particular,
it (hopefully) dispenses with the path problem.
Before anyone asks, I'm not tied to any of the command names or the option
names that I've used here, nor am I particularly tied to the XML examples;
as long as we capture the content properly I don't care what the element
names are :-)
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Dan M. <d-...@uc...> - 2000-11-20 21:53:04
|
Nik,
WOW!
This is impressive. I read about 60% of it and then decided to print it
out for a closer look and so I can keep notes. So far it looks really
good.
Dan
On Mon, 20 Nov 2000, Nik Clayton wrote:
> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
>
> There should be no such thing as a non-relocatable package.
>
> By all means, allow default paths to be included, but they must always
> be overridable at installation time by the admin.
>
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
>
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
>
> 2. Maintaining a TOC from the Contents List
>
> 3. Maintaining an Index from the Contents List
>
> 4. Maintaining a Categories list
>
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
>
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
>
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
>
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
>
> 2. Locally installed system documentation, installed by the SA team.
>
> 3. Locally installed documentation, installed by the user.
>
> 4. Remote documentation on a web site.
>
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
>
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
>
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
>
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
>
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
>
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
>
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
>
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
>
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
>
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
>
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
>
> The OMF information may not even be on the same host as the target
> document.
>
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
>
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
>
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
>
> sk should treat this as one document with three formats, rather than
> three separate documents.
>
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
>
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
>
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
>
> -db The Contents List to update. Uses a default if not specified.
>
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
>
> This command does not physically copy the file in to place.
>
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
>
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
>
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
>
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
>
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
>
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
>
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
>
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
>
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
>
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
>
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
>
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
>
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
>
> Normally, this behaviour would be carried out by pkg_delete, or whatever
> packaging system the user is using.
>
> Querying installed documentation
>
> There should probably be a command line tool to allow the user to do
> simple queries of the Contents List. Since the Contents List is in XML,
> any XML aware tool will be able to handle it. However, sometimes you
> just want to use awk.
>
> sk-query -format csv
>
> would dump the Contents List as CSV format.
>
> id,name,format,path,...
>
> Since the data is multi-dimensional, you would probably end up with
> multiple lines with the same id and name, but different formats and paths.
>
> It should be possible to query individual document IDs to retrieve
> information about them;
>
> sk-query -id MD5
>
> and probably, given a pathname to a document, retrieve information about
> it as well,
>
> sk-query -path /usr/local/share/doc/app-name/html/index.html
>
> Sanity checking the Contents List
>
> There should be a tool to sanity check the Contents List, and warn the
> end user of potential problems.
>
> Several that spring immediately to mind.
>
> 1. Where a <docinstance> element exists, but for which no corresponding
> files are found on the file system.
>
> 2. Files for which a <docinstance> element exists which points to
> a remote document that can no longer be found.
>
> 3. I glossed over it earlier, but each <docinstance> should have a
> modtime attribute, that contains the file's modification time.
> Part of the sanity check should ensure that the modtime attribute
> and the files (or the remote files) modification times do match.
> If they don't, it's likely that the OMF data and/or the TOC and Index
> are out of date, and need to be regenerated.
>
> None of these are fatal errors, but they will cause the generated TOC and
> Index to potentially be wrong. The tools should have options to fix the
> errors themselves (in the case of (1) and (2)).
>
> I hope all that makes sense. It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer. In particular,
> it (hopefully) dispenses with the path problem.
>
> Before anyone asks, I'm not tied to any of the command names or the option
> names that I've used here, nor am I particularly tied to the XML examples;
> as long as we capture the content properly I don't care what the element
> names are :-)
>
> N
|
|
From: Laszlo K. <las...@su...> - 2000-11-21 10:08:45
|
Nik Clayton wrote: > > On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote: > > [snip] > > > There should be no such thing as a non-relocatable package. > > > > I guess what we are trying to do is to create support for both > > relocatable and non-relocatable packages. It is up to the developer and > > the packager to decide if and how they want to use this support. > > There should be no such thing as a non-relocatable package. > Are you saying that we shouldn't offer the choice I outlined above and we should force people to build fully relocatable packages (the documentation part at least) even if they don't want to? Laszlo |
|
From: Nik C. <ni...@no...> - 2000-11-21 17:57:22
|
On Tue, Nov 21, 2000 at 10:08:42AM +0000, Laszlo Kovacs wrote: > Nik Clayton wrote: > > > > On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote: > > > [snip] > > > > There should be no such thing as a non-relocatable package. > > > > > > I guess what we are trying to do is to create support for both > > > relocatable and non-relocatable packages. It is up to the developer and > > > the packager to decide if and how they want to use this support. > > > > There should be no such thing as a non-relocatable package. > > > Are you saying that we shouldn't offer the choice I outlined above and > we should force people to build fully relocatable packages (the > documentation part at least) even if they don't want to? Yes. The idea of being told "You can only install this document on your system if you install it _here_" is completely bizarre to me. N -- Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95. Telephone line, $24.95 a month. Software, free. USENET transmission, hundreds if not thousands of dollars. Thinking before posting, priceless. Somethings in life you can't buy. For everything else, there's MasterCard. -- Graham Reed, in the Scary Devil Monastery |
|
From: Nik C. <ni...@no...> - 2000-11-21 12:56:59
|
Chaps,
Two things, while I remember.
First;
On Mon, Nov 20, 2000 at 09:20:47PM +0000, Nik Clayton wrote:
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
<snip>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
>
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
Done properly, this should be trivial, using XSLT.
Second;
/var/db/master.xml (or whatever it is) should not (contrary to my message)
handle the grouping of documents in to categories. By that I mean that
it shouldn't do
<sect>
<title>Apps</title>
<sect>
<title>Xemacs</title>
<doc>...</doc>
<doc>...</doc>
<doc>...</doc>
</sect>
...
</sect>
Why? Because then it's a pain in the backside to have the same document
appear in multiple categories, as you have to list it twice. Instead,
the format should be more like:
<doc docid="...">
<docinstance .../>
<category>Applications</category>
<category>Editors</category>
<category>Lisp</category>
</doc>
This can then be arranged in to groupings by the same process that generates
the TOC and Index.
Supporting a hierarchy of categories is an exercise left for the reader :-)
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Laszlo K. <las...@su...> - 2000-11-21 13:36:40
|
[snip] > > Second; > > /var/db/master.xml (or whatever it is) should not (contrary to my message) > handle the grouping of documents in to categories. By that I mean that > it shouldn't do > > <sect> > <title>Apps</title> > <sect> > <title>Xemacs</title> > > <doc>...</doc> > <doc>...</doc> > <doc>...</doc> > </sect> > > ... > </sect> > > Why? Because then it's a pain in the backside to have the same document > appear in multiple categories, as you have to list it twice. Instead, > the format should be more like: > > <doc docid="..."> > <docinstance .../> > > <category>Applications</category> > <category>Editors</category> > <category>Lisp</category> > </doc> > > This can then be arranged in to groupings by the same process that generates > the TOC and Index. Actually it already works in the first way you described above. And I don't find it hard at all to manage it:-) Laszlo |
|
From: Nik C. <ni...@no...> - 2000-11-21 17:16:10
|
On Tue, Nov 21, 2000 at 01:36:34PM +0000, Laszlo Kovacs wrote:
> > /var/db/master.xml (or whatever it is) should not (contrary to my message)
> > handle the grouping of documents in to categories. By that I mean that
> > it shouldn't do
> >
> > <sect>
> > <title>Apps</title>
> > <sect>
> > <title>Xemacs</title>
> >
> > <doc>...</doc>
> > <doc>...</doc>
> > <doc>...</doc>
> > </sect>
> >
> > ...
> > </sect>
> >
> > Why? Because then it's a pain in the backside to have the same document
> > appear in multiple categories, as you have to list it twice. Instead,
> > the format should be more like:
> >
> > <doc docid="...">
> > <docinstance .../>
> >
> > <category>Applications</category>
> > <category>Editors</category>
> > <category>Lisp</category>
> > </doc>
> >
> > This can then be arranged in to groupings by the same process that generates
> > the TOC and Index.
>
> Actually it already works in the first way you described above. And I
> don't find it hard at all to manage it:-)
I think that's going to cause problems.
Consider Postgres. That's going to install FAQs, Manuals, man pages (as
distinct from a Manual or User Guide), README files, and so on, and so
forth.
Under your scheme, assume that we have a FAQ, Manuals, Man page (1), Man
page (2), Man page ..., and README categories, as well as a category for
Postgres.
Now, each Postgres doc is going to appear once in the Postgres category.
Then, some of those docs are going to be duplicated in the FAQ category,
some in the README category, some in the "Man Page (1)" category, and so
on.
This going to be even worse when you install a big Gnome application,
and you want to make sure that its documents are also in the Gnome
category.
I think all this duplication is a bad thing, and would much rather see
something like
<doc docid="...">
<docinstance url="file:/usr/local/share/man/man1/psql.1">
<!-- Other elements ommitted -->
<category>Postgres</category>
<category>man(1)</category>
</doc>
Than
<doc docid="...">
<docinstance url="file:/usr/local/share/man/man1/psql.1">
<category>Postgres</category>
</doc>
<doc docid="...">
<docinstance url="file:/usr/local/share/man/man1/psql.1">
<category>man(1)</category>
</doc>
Or,
<category>
<title>Postgres</title>
<doc id="...">
<!-- Ommitted -->
</doc>
</category>
<category>
<title>man(1)</title>
<doc id="...">
<!-- Ommitted -->
</doc>
</category>
This is basic information normalisation theory.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Laszlo K. <las...@su...> - 2000-11-21 13:34:50
|
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
Yes, but we didn't aim for more than one.
> 2. Maintaining a TOC from the Contents List
Yes.
> 3. Maintaining an Index from the Contents List
Yes.
> 4. Maintaining a Categories list
What is the difference between this and (1)?
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
>
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
Multiple content lists would probably be a lower priority.
>
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
>
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
It will be able to categorize them if OMF files are created for them. I
think sk wants to categorize docs based on OMF files while the ones
without OMF files would not be considered by it. They can be considered
by the help browser though as it happens now with Nautilus in Gnome.
> 2. Locally installed system documentation, installed by the SA team.
>
> 3. Locally installed documentation, installed by the user.
>
> 4. Remote documentation on a web site.
Yes to all.
>
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
As I said before sk works with one Content List at moment and we did not
think about more than one. You might want to detail how you imagine
this.
>
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
>
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
>
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
Waht do you mean about TOC update? At the moment this is extracted at
install time (from sgml only) and stored in a file. As long as the doc
does not change, the TOC does not need to be updated.
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
Saem as for TOC above,
>
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
The sk Content List is XML, why would we want to generate HTML? I guess
it depends on the browser what kind of files are needed. Anyway I am
sure our XML Content List can be easily converted to HTML.
>
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
>
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
This has to be discussed. OMF is the heart of Scrollkeeper, I am not
sure we intend to make Scrollkeeper to deal with docs that don't have
OMFs. But as I said before, the browser can deal with them separately.
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
Yes, that is a good idea.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
>
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
This is planned for SGML, it can be done for other formats too.
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
Yes, this is how it works.
> The OMF information may not even be on the same host as the target
> document.
At the moment the OMF information is local, the document might not be.
However the current implementation supports only local docs.
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
Currently we take for granted that the OMF is changed at any doc change.
If the doc has an OMF then when the whole package is re-installed then
the OMF changes (the timestamp, this is what we monitor).
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
I think this is working. You can certainly install a doc in multiple
places in the Content List.
>
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
>
> sk should treat this as one document with three formats, rather than
> three separate documents.
>
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
It doesn't work like this right now, every doc needs a separate OMF
file. However one OMF file with several doc entries each describing a
different doc is supported. So it would be possible to use one OMF file
for the same doc in multiple formats.
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
>
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
>
> -db The Contents List to update. Uses a default if not specified.
>
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
>
> This command does not physically copy the file in to place.
I guess here you mean that most or all of the OMF metadata should be
able to be passed in as a parameter in the installation script/binary.
Good idea.
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
md5 was ruled out before because it is too slow on Solaris. But we have
a mechanism of assigning unique IDs by storing the assigned ones and
then just not reassigning them again.
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
>
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
The current implementation assigns different IDs for the same doc in
different formats. However I don't see this as a problem as when we
uninstall docs the OMF files are deleted, scrollkeeper then uninstall
all the related docs.
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
That sounds good.
>
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
>
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
>
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
The current process is much simpler than this and it achieves the same
thing. Maybe you should also look in more detail to the current
implementation.
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
>
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
>
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
>
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
All of this can be implemented, but I don't see it too important for the
moment.
>
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
>
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
I guess we can support this later on.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
>
> Normally, this behaviour would be carried out by pkg_delete, or whatever
> packaging system the user is using.
This is fully automatic, if you delete the OMF file, all the related
docs will be uninstalled.
>
> Querying installed documentation
>
> There should probably be a command line tool to allow the user to do
> simple queries of the Contents List. Since the Contents List is in XML,
> any XML aware tool will be able to handle it. However, sometimes you
> just want to use awk.
>
> sk-query -format csv
>
> would dump the Contents List as CSV format.
>
> id,name,format,path,...
>
> Since the data is multi-dimensional, you would probably end up with
> multiple lines with the same id and name, but different formats and paths.
>
> It should be possible to query individual document IDs to retrieve
> information about them;
>
> sk-query -id MD5
>
> and probably, given a pathname to a document, retrieve information about
> it as well,
>
> sk-query -path /usr/local/share/doc/app-name/html/index.html
Good idea.
>
> Sanity checking the Contents List
>
> There should be a tool to sanity check the Contents List, and warn the
> end user of potential problems.
>
> Several that spring immediately to mind.
>
> 1. Where a <docinstance> element exists, but for which no corresponding
> files are found on the file system.
>
> 2. Files for which a <docinstance> element exists which points to
> a remote document that can no longer be found.
>
> 3. I glossed over it earlier, but each <docinstance> should have a
> modtime attribute, that contains the file's modification time.
> Part of the sanity check should ensure that the modtime attribute
> and the files (or the remote files) modification times do match.
> If they don't, it's likely that the OMF data and/or the TOC and Index
> are out of date, and need to be regenerated.
>
> None of these are fatal errors, but they will cause the generated TOC and
> Index to potentially be wrong. The tools should have options to fix the
> errors themselves (in the case of (1) and (2)).
Good idea.
> I hope all that makes sense. It's not too different from what's gone before,
> but I hope the extra verbiage makes things a little clearer. In particular,
> it (hopefully) dispenses with the path problem.
Yes it helps a lot. You look like having lots of experience in the area.
Laszlo
|
|
From: Nik C. <ni...@no...> - 2000-11-21 16:17:52
|
On Tue, Nov 21, 2000 at 01:34:42PM +0000, Laszlo Kovacs wrote:
> > 1. Maintaining one or more lists of documentation that the user wants
> > a TOC for (the "Contents List" from the most recent proposal)
>
> Yes, but we didn't aim for more than one.
I think we need more than one. I explain why in a bit.
> > 4. Maintaining a Categories list
>
> What is the difference between this and (1)?
Depends on whether or not sk enforces the list of categories that
documentation can be in or not. If it doesn't, then no list.
Pro: Everything's very easy.
Con: If the end user makes a typo when choosing a category for the
document, sk can't warn them about it.
> Multiple content lists would probably be a lower priority.
>
> >
> > What sort of documentation can the Contents List going to point to?
> >
> > 1. Locally installed standard documentation that comes with the system,
> > such as man pages, or Info documents.
> >
> > QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> > these?
>
> It will be able to categorize them if OMF files are created for them. I
> think sk wants to categorize docs based on OMF files while the ones
> without OMF files would not be considered by it. They can be considered
> by the help browser though as it happens now with Nautilus in Gnome.
I think this is a bad idea. If you go this route then sk becomes another
format, like info, or man, or html, and the help browsers have to deal with
this, and still need to go and handle the rest of the documentation in the
system.
I think sk has a much better chance of being adopted if it is very easy for
legacy documentation to be 'adopted' by sk, so that, at the very least, the
documentation makes it in to the Contents List. Producing detailed OMF for
legacy documentation can be done after the document has been adopted.
DD: Legacy documentation (with, in the worst case, just a filename) can
be adopted by SK so that it appears in the Contents List. The end user
won't be able to do very much with it until there's meaningful OMF for
the legacy docs, but it's a start, and should assist users who want to
write OMF for legacy docs.
<snip>
> > I think it is fair to say that, when run, sk will need to construct it's
> > global Contents List by parsing multiple, smaller, contents lists. In much
> > the same way that SGML catalog files can include one another at the moment.
>
> As I said before sk works with one Content List at moment and we did not
> think about more than one. You might want to detail how you imagine
> this.
The SA generates a contents list for the host, call it /var/db/sk/contents.xml
or something (I was calling it master.xml in the previous examples).
A regular user has also installed some documentation under $HOME, and have
created $HOME/.sk/db/contents.xml
The user wants to create TOC and Index files, personal to them, that contain
both these lists.
Either of the following approaches should work (to generate a personal TOC);
1. Use multiple '-d' parameters (or -c, or whatever)
sk-generate-toc -d /var/db/sk/contents.xml -d ~/.sk/db/contents.xml
2. Use an environment variable, which contains a ':' delimited list of
files to search.
SK_CONTENTS=/var/db/contents.xml:~/.sk/db/contents.xml sk-generate-toc
3. Create a meta-contents file, that uses entity inclusions to pull in the
files. Something like this;
<!DOCTYPE contents PUBLIC "-//ScrollKeeper Contents List 1.0//EN" [
<!ENTITY systemwide SYSTEM "/var/db/sk/contents.xml">
<!ENTITY personal SYSTEM "/home/nik/.sk/db/contents.xml">
]>
<contents>
&systemwide;
&personal;
</content>
[ I've assumed here that XML can do this in the same way that you can
with SGML. I know more SGML than I do XML, so this might not be
the case. ]
And then point sk-generate-toc at this meta-contents file;
sk-generate-toc -d meta-contents.xml
> > DD: Contents Lists must be able to refer to other Contents Lists, and sk
> > must have an option to chase the link to read the referred to Contents
> > List.
> >
> > DD: sk must have a run time option to allow the end user to select which
> > Contents Lists the user wants sk to process.
> >
> > DD: sk must have a run time option to allow the end user to select which
> > TOC to update.
>
> Waht do you mean about TOC update? At the moment this is extracted at
> install time (from sgml only) and stored in a file. As long as the doc
> does not change, the TOC does not need to be updated.
Joe Sysadmin runs sk-generate-toc to generate a TOC suitable for the entire
host, and puts it in /usr/local/share/doc/TOC
Joe User runs sk-generate-toc to generate a TOC that includes documentation
that they've installed documentation under $HOME.
Just so I'm clear here -- I'm not talking about the TOC that gets stored in
the Contents List. I am assuming that the TOC will need to be extracted
from the Contents List in other formats (such as HTML) to support environments
where KDE or GNOME are not the norm.
It would be nice to assume that everybody who is going to run sk is going
to be running a desktop environment that supports these things, but not
everybody is. I also suspect that a lot of people are going to need little
more than a mechanism which generates an HTML ToC and Index for their
installed documentation.
> > DD: sk must have a run time option to allow the end user to select which
> > Index to update.
>
> Saem as for TOC above,
As above.
> > DD: One of the sk commands must be to generate a usable Contents List
> > (which really means {X}HTML version, to start with) given a Contents
> > List which is in sk internal format.
>
> The sk Content List is XML, why would we want to generate HTML? I guess
> it depends on the browser what kind of files are needed. Anyway I am
> sure our XML Content List can be easily converted to HTML.
As above.
> > Each piece of documentation (which might really consist of tens or even
> > hundreds of files) can be considered to consist of two parts.
> >
> > 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
> >
> > 2. The OMF, containing the 17 (?) key pieces of meta information that
> > sk likes to keep about the documentation. Not all 17 pieces of
> > information may necessarily be present.
> >
> > Not all documentation the user wants to point to is going to have the OMF
> > information associated with it. This is unavoidable.
> >
> > DD: sk must work tolerably well with minimum OMF information. At the
> > worst case, sk must do something useful when the only OMF information
> > available is the document title, and a URL pointing to the document's
> > 'index' page.
> >
> > Actually, I take that back. The only piece of information sk needs
> > is the URL. In the worst case (with 0 OMF information) the URL
> > becomes the title, until the end user provides some OMF, or points
> > sk at pre-existing OMF.
>
> This has to be discussed. OMF is the heart of Scrollkeeper, I am not
> sure we intend to make Scrollkeeper to deal with docs that don't have
> OMFs. But as I said before, the browser can deal with them separately.
I know. All I'm saying is that in the worst case (when a document has been
'adopted' by SK, but with no OMF, SK should synthesise an OMF that contains
1 piece of information, the document's title, which is based on the filename.
> > DD: sk should probably include some support applications which can parse
> > common documentation formats (*roff man and mdoc, GNU info, HTML,
> > LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> > them as possible.
>
> Yes, that is a good idea.
We can call them plugins. That's a nice, sexy, buzzwordy term :-)
> > In some cases this is necessary to produce bare bones OMF files that
> > will not be processed further. In other cases, this will be to produce
> > OMF files that will be further customised and improved by the end user.
> >
> > DD: sk should include some support applications that can generate TOC and
> > Index information from a variety of source formats.
>
> This is planned for SGML, it can be done for other formats too.
More plugins :-)
> > DD: sk should not assume that the OMF information is in the same directory
> > as the document, nor should it assume that the location of the OMF
> > information can be inferred based on the document's location. The
> > location of the OMF information must always be explicitly listed in
> > the Contents List.
>
> Yes, this is how it works.
Thought so.
> > The OMF information may not even be on the same host as the target
> > document.
>
> At the moment the OMF information is local, the document might not be.
> However the current implementation supports only local docs.
That's fine. I just want to make sure that these design decisions are
written down somewhere. Ideally, we then have a list of things to implement,
which can be checked off one by one.
> > DD: It is possible for a document to change, but for the OMF information
> > to be neglected. sk should maintain a modification time stamp for
> > each document and its corresponding OMF, to allow the end user to
> > determine when that might have happened, and to correct for it.
>
> Currently we take for granted that the OMF is changed at any doc change.
> If the doc has an OMF then when the whole package is re-installed then
> the OMF changes (the timestamp, this is what we monitor).
A document won't always be installed with new OMF. For example, if SK is
maintaining information about a remote document, but with the OMF locally.
The remote document might change, and SK needs to be able to tell when the
OMF is out of date (timestamps) with respect to the document.
> > DD: An application may have multiple pieces of documentation associated
> > with it. Applications are also not the only things that have
> > documentation, the system does (think: man page sections). Simply
> > storing information about a document isn't enough, you need to be
> > able to group documents into chunks (and probably group chunks into
> > chunks, or allow the same document to appear in multiple categories).
> > I'm not going to discuss this in too much detail in this document, this
> > is the reason for "4. Maintaining a Categories list" earlier.
>
> I think this is working. You can certainly install a doc in multiple
> places in the Content List.
I don't think we want to install the same document in multiple places.
Apart from ballooning the size of the Contents List, it means that there
is greater chance that one or more of those copies will be neglected during
an up date.
Each document should only appear once in the Contents List, but SK should be
able to handle multiple categories assigned to the same document.
> > DD: One document, multiple output formats. The user might have installed
> > the same document three times, once as HTML, once as PDF, and once as
> > plain text. The three installations are the same document, and have
> > the same OMF associated with them.
> >
> > sk should treat this as one document with three formats, rather than
> > three separate documents.
> >
> > [ This is probably going to be the most contentious piece of this
> > writeup, and it might change. In particular, it also suggests that
> > translations of a document should be treated as the same document,
> > just in a different language, and I'm not sure that's a good idea.
> > Then again, it might be. ]
>
> It doesn't work like this right now, every doc needs a separate OMF
> file. However one OMF file with several doc entries each describing a
> different doc is supported. So it would be possible to use one OMF file
> for the same doc in multiple formats.
This is a pretty crucial decision to make, and I haven't seen it discussed
on the list.
Gut instinct tells me that it's a cleaner design to consider that is 1:M
mapping between documents, and instances, instead of a 1:1 mapping.
I'll try and come up with some convincing reasons for this.
> > SK Tasks
> > --------
> >
> > There are a number of tasks that the sk end user is going to want to tell
> > sk to do. They are;
> >
> > Install new documentation that comes bundled with OMF
<snip>
> > 'sk-install-doc' registers the document. The two (optional) arguments
> > here are
<snip>
> > This command does not physically copy the file in to place.
>
> I guess here you mean that most or all of the OMF metadata should be
> able to be passed in as a parameter in the installation script/binary.
> Good idea.
No. At this point, no OMF information has been installed.
Installing the OMF for this document is a separate task.
[ Re MD5 ]
> md5 was ruled out before because it is too slow on Solaris. But we have
> a mechanism of assigning unique IDs by storing the assigned ones and
> then just not reassigning them again.
This is implementation specific, and not really important.
<snip>
> The current implementation assigns different IDs for the same doc in
> different formats.
This comes back to the earlier point, about a 1:M or 1:1 mapping. This
needs to be decided.
<snip>
> All of this can be implemented, but I don't see it too important for the
> moment.
I do, which is why I'm bringing it up. I hadn't seen anything talking about
SK from a user's point of view, or the sort of tasks a user is going to
want to carry out. I'd like to hammer out some sort of specification that
we can try and agree on, and codeto.
> > Install new documentation that doesn't come with OMF
> >
> > This is very similar to the previous example. However, instead of
> > running sk-install-omf, the user has two choices.
<snip>
> I guess we can support this later on.
We should be able to support this out of the box. Remember that if the
document has no OMF supplied by the end user then (IMHO) SK should synthesise
some OMF.
> > Removing documentation
> >
> > If the user wants to remove a single document format, but keep the
> > information about the document, they might do
> >
> > sk-remove-doc -format pdf -id MD5
> >
> > which would remove the <docinstance> information for the PDF version of
> > the document with id == MD5.
> >
> > If they want to remove a complete document, they do
> >
> > sk-remove-doc -id MD5
> >
> > which removes the whole thing. This does not touch the document's files
> > (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> > List, and if the OMF has been stored in a separate file then it removes
> > that as well.
> >
> > Normally, this behaviour would be carried out by pkg_delete, or whatever
> > packaging system the user is using.
>
> This is fully automatic, if you delete the OMF file, all the related
> docs will be uninstalled.
I might not want to do that though. I might be deleting the OMF file because
it's out of date, and I haven't got time to update it.
As I say, in the absence of OMF, SK should fake some reasonable OMF, and
keep going.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Dan M. <d-...@uc...> - 2000-11-21 22:39:33
|
On Mon, 20 Nov 2000, Nik Clayton wrote:
> On Mon, Nov 20, 2000 at 02:48:25PM +0000, Laszlo Kovacs wrote:
> > [snip]
> > > There should be no such thing as a non-relocatable package.
> >
> > I guess what we are trying to do is to create support for both
> > relocatable and non-relocatable packages. It is up to the developer and
> > the packager to decide if and how they want to use this support.
>
> There should be no such thing as a non-relocatable package.
Both relocatable and non-relocatable packages exist, so scrollkeeper
should support them both in the simplest possible way.
> By all means, allow default paths to be included, but they must always
> be overridable at installation time by the admin.
>
> Imagine an RPM that wanted to install in /opt, and wouldn't let you install
> it anywhere else.
>
> Based on the design docs on the web, what does sk have responsibility for:
>
> [ This is my current understanding. Please correct any misconceptions I
> might have. Things I flag with 'DD' are design decisions that I think
> fall out of the model. ]
>
> 1. Maintaining one or more lists of documentation that the user wants
> a TOC for (the "Contents List" from the most recent proposal)
The current way things are designed is that it will have one contents list
for the system. Of course you can always create another one somewhere
else using the -p flag, but generally they would contain the same
information and only the default one will get updated each time a document
is installed. By specifying which OMF directories you want to include,
these databases could be made to differ. We don't currently have a
mechanism for supporting or specifying multiple OMF directories, but that
would be fairly straightforward and is planned. This will make it very
simple for users to have their own local databases which vary from the
system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
which add to or over-ride the system defaults.
It is not clear how and why one would want to use multiple contents lists
except for individual users and/or hand customization. The problem is
that somebody has to specify what each database contains. So it works for
users, because they can hand-tweak the contents. It doesn't work very
well for system-wide databases, since everything must be specified by the
packager. So while you could imagine GNOME documents being kept in one
list and KDE documents being kept in another list, this doesn't really
achieve anything. The information which would be passed on the command
line to sk to indicate which list the doc goes in is really metadata which
belongs in the OMF. OMF and XML provide everything we need to have one
database and understand the scope of each piece. (eg. whether it is a
GNOME doc or KDE doc.)
Even in my example of users having their own configuration, we are never
merging contents list. In this case the user's personal database is built
directly from OMF files just like the system-wide database. Doing it
otherwise would make it almost impossible for a user to over-ride the
categorization of the system-wide database, over-ride or remove particular
entries in the system-wide database, or merge a doc into a part of the
database which is typically in the system-wide database. Plus it is just
harder to do and messy.
Am I overlooking some particularly good reason to try to merge contents
list?
What is the advantage of breaking the main database into pieces?
> 2. Maintaining a TOC from the Contents List
>
> 3. Maintaining an Index from the Contents List
>
> 4. Maintaining a Categories list
Well, this is somewhat debatable and worthy of discussion. Somebody has
to provide and maintain the categories list. On one side, we can just
define it as part of OMF/sk and install a static list with sk. Then no
maintenance is necessary. Or, we may consider having this specified by a
configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
sys admin could drop in a replacement if they want. For example you could
chop out all the HOWTO's and man pages if you know newbies who never touch
a command line will be using your machine.
> [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> application will have the responsibility for handling user queries of the
> SK TOC and Index files ]
sk should export searching functionality. Basic metadata searches can be
implemented from scratch in sk. We may want to rely on an external engine
for more complex searches potentially, but the help browser shouldn't have
to know or care about that. It should interface directly with sk.
> Activities (2) and (3) require a Contents List. Note that a don't say "the
> {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> maintain several different TOCs and Indices from several different Contents
> Lists on the same host (for example, a regular user might want to run sk
> to maintain their own index and TOC of ~/my-docs).
For a given database, I agree. Eventually, I see the system having one
database (with one contents list) and possibly each user having one. Is
this what you have in mind? If you think we want more than one contents
list in the system's database, please explain why.
> What sort of documentation can the Contents List going to point to?
>
> 1. Locally installed standard documentation that comes with the system,
> such as man pages, or Info documents.
Perhaps. I think we probably do want to include this information, using
automagically generated (and therefore incomplete) OMF metadata as you
suggest below. For man and info, we would have to update this information
off of cron unfortunately. So there will be an unavoidable delay between
when a document is installed and when it is registered with sk.
> QUESTION: Is sk going to deal with non-SGML/XML documents, such as
> these?
Definitely. It should support arbitrary doc formats. For some document
formats (esp. DocBook/XML), it can extract some nice info like TOC and
Index. In fact, it will eventually be able to extract the whole OMF from
DocBook/XML docs, as Norman Walsh agreed to include any missing tags.
> 2. Locally installed system documentation, installed by the SA team.
SA =? sys admin
> 3. Locally installed documentation, installed by the user.
Yes. Although like much of this discussion, this feature will come in a
post-0.1 release.
> 4. Remote documentation on a web site.
Yes.
> I think it is fair to say that, when run, sk will need to construct it's
> global Contents List by parsing multiple, smaller, contents lists. In much
> the same way that SGML catalog files can include one another at the moment.
Why? It sounds very nice and wonderful. However, this adds a layer of
complexity without any obvious benefits.
> DD: Contents Lists must be able to refer to other Contents Lists, and sk
> must have an option to chase the link to read the referred to Contents
> List.
(as above)
> DD: sk must have a run time option to allow the end user to select which
> Contents Lists the user wants sk to process.
Yes. If we have seperate databases under $localstatedir and $HOME, the
user should be able to (through the help browser configuration
dialog) configure which database to use.
> DD: sk must have a run time option to allow the end user to select which
> TOC to update.
The databases are stored together. So, this should be specified at the
same time as the contents list part of the database. I don't think I want
my contents list coming from one database and my toc from another.
(Or am I misunderstanding you? I'm not sure what you mean by "update" a
TOC.)
> DD: sk must have a run time option to allow the end user to select which
> Index to update.
(as above)
> DD: One of the sk commands must be to generate a usable Contents List
> (which really means {X}HTML version, to start with) given a Contents
> List which is in sk internal format.
Everything is XML right now, so this is not an issue. If we decide
another internal format is better we should still export XML to the
browser. As for HTML, this is possible and may be a nice interface for
people who want to create a very minimalistic web browser.
> Each piece of documentation (which might really consist of tens or even
> hundreds of files) can be considered to consist of two parts.
>
> 1. The documentation itself (*.html, *.png, foo.pdf, whatever).
>
> 2. The OMF, containing the 17 (?) key pieces of meta information that
> sk likes to keep about the documentation. Not all 17 pieces of
> information may necessarily be present.
Yes. We will have to look closely at how sk handles various pieces of
data missing.
> Not all documentation the user wants to point to is going to have the OMF
> information associated with it. This is unavoidable.
>
> DD: sk must work tolerably well with minimum OMF information. At the
> worst case, sk must do something useful when the only OMF information
> available is the document title, and a URL pointing to the document's
> 'index' page.
>
> Actually, I take that back. The only piece of information sk needs
> is the URL. In the worst case (with 0 OMF information) the URL
> becomes the title, until the end user provides some OMF, or points
> sk at pre-existing OMF.
Good idea.
> DD: sk should probably include some support applications which can parse
> common documentation formats (*roff man and mdoc, GNU info, HTML,
> LinuxDoc, LaTeX, DocBook) and extract as much OMF information from
> them as possible.
>
> In some cases this is necessary to produce bare bones OMF files that
> will not be processed further. In other cases, this will be to produce
> OMF files that will be further customised and improved by the end user.
Yes :)
> DD: sk should include some support applications that can generate TOC and
> Index information from a variety of source formats.
yes.
> DD: sk should not assume that the OMF information is in the same directory
> as the document, nor should it assume that the location of the OMF
> information can be inferred based on the document's location. The
> location of the OMF information must always be explicitly listed in
> the Contents List.
>
> The OMF information may not even be on the same host as the target
> document.
Yes. (This is in the existing design.)
> DD: It is possible for a document to change, but for the OMF information
> to be neglected. sk should maintain a modification time stamp for
> each document and its corresponding OMF, to allow the end user to
> determine when that might have happened, and to correct for it.
For the current version of sk, which does not yet allow for remote
documents, only the OMF file's timestamp is watched. This is because if
somebody edits the doc and not the OMF, then it is their fault for not
updating the OMF, not sk's. In reality whenever you install a new version
of a package, even if the OMF is not updated (as it should), it still
updates the old version so it gets a new timestamp and the new doc gets
registered despite any negligence on the part of the maintainer to update
the OMF (or the doc for that matter;).
This mainly becomes an issue for:
1) remote docs
2) docs which users write and modify themselves.
So we may indeed need to start tracking the doc timestamps when we support
these features. We have to be careful about constantly checking the
timestamps on a bunch of docs which are spread around the internet
though. There will be many cases where this task cannot complete, or else
takes an unacceptable amount of time.
> DD: An application may have multiple pieces of documentation associated
> with it. Applications are also not the only things that have
> documentation, the system does (think: man page sections). Simply
> storing information about a document isn't enough, you need to be
> able to group documents into chunks (and probably group chunks into
> chunks, or allow the same document to appear in multiple categories).
> I'm not going to discuss this in too much detail in this document, this
> is the reason for "4. Maintaining a Categories list" earlier.
That is why we have a contents list and categories file. This is all
specified by OMF metadata.
> DD: One document, multiple output formats. The user might have installed
> the same document three times, once as HTML, once as PDF, and once as
> plain text. The three installations are the same document, and have
> the same OMF associated with them.
No. These definitely do not have the same OMF, unless they are identical
copies. The OMF file specifies the document format in FORMAT.DTD and
FORMAT.MIME. The docs also have different IDENTIFIER's.
(We may decide to expand this a little bit. FORMAT.STYLESHEET may be
nice, as it would allow the doc authors to dictate the appearance of their
doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
icon, etc.)
> sk should treat this as one document with three formats, rather than
> three separate documents.
If you look in the ldp-meta archives from several months ago, I proposed
we add a document id to the OMF. This would allow us to identify if two
documents are "The-Same". I call two documents "The-Same" if they are the
same document only different versions, languages, formats, etc. If two
documents have the same title, they are not necessarily
"The-Same". Likewise, if I change the title of my document at some point
in time, I can have two docs which are "The-Same" but not having the same
title.
Obviously this sort of information will be very nice to have. You can do
things like "Get me this document in language xxx." or "Show me the latest
version of this document on the Net." or "Show me the previous version of
this document from the Net."
> [ This is probably going to be the most contentious piece of this
> writeup, and it might change. In particular, it also suggests that
> translations of a document should be treated as the same document,
> just in a different language, and I'm not sure that's a good idea.
> Then again, it might be. ]
I guess we were thinking along similar lines :)
> SK Tasks
> --------
>
> There are a number of tasks that the sk end user is going to want to tell
> sk to do. They are;
>
> Install new documentation that comes bundled with OMF
>
> The user has just downloaded and installed a package that comes with one
> or more bundled documents, and the package maintainer has thoughtfully
> provided an OMF file for each document.
>
> Each document has been provided in HTML and PDF format.
>
> Suppose the user installs the HTML documentation first, and installs it
> it into /usr/local/share/doc/app-name/html/, where normal file is called
> index.html.
>
> [ Everywhere I say "the user runs" I really mean "The package maintainer
> does this by means of a post-install script that they've already written.
> It is assumed that this post-install script knows the directories that
> documentation has been installed in, by virtue of being part of the
> package that did the installation in the first place. ]
>
> The user runs
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml \
> /usr/local/share/doc/app-name/html/index.html
Why do we specify the format on the command line? The FORMAT is specified
in the OMF already. The packager should just install and register the
metadata instead of putting it on the command line.
> 'sk-install-doc' registers the document. The two (optional) arguments
> here are
>
> -format The format of the installed documentation. Might be
> html-split - Bunch of small HTML files
> html - One big HTML file
> ps - postscript
> pdf - PDF
> ... - extend as necessary (rtf, txt, pdb, ...)
>
> If not specified, sk can try and guess what it is.
I don't think we really want this. One could make an argument that if no
OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
has a bunch of arguments for each element in the OMF. However I'm not
sure if we want to merge all this into the same command that registers the
document with sk.
> -db The Contents List to update. Uses a default if not specified.
Here I assume (hope?;) that you mean the database directory and not the
contents list. If so, we already do this with the "-p" flag.
> [ Note: This also needs a category option, so that the end user can
> specify which categories the document goes in to. By default,
> it goes in to "unfiled". ]
Here we go, implementing the OMF on the command-line ;)
We should put all this metadata in the OMF metadata file.
> This command does not physically copy the file in to place.
>
> sk-install-doc should generate a unique identifier for this document
> (MD5 hash of the filename?), and display this to the user. We probably
> also need a standalone utility (sk-generate-id?) that, given a path,
> generates an ID.
Right now we just number them, which guarantees a unique identifier. We
used to have functions similar to the latter one, but decided to removed
them. This was because we should not encourage the help browser to
actually use these id's since they may be modified when applications are
uninstalled or installed which could break a running help browser which
doesn't realize these changes have occured.
> At this point, /var/db/sk/master.xml looks something like this:
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
I think this file should hold information taken from the OMF file, as
described in the previous proposals. Duplicating OMF information with
command-line information is redundant, puts more burden on the packager,
and increases the chance of mistakes and conflicts occuring.
> As you can see, not a lot of information in there so far. "lang" and
> "encoding" could also be specified on the command line, and if not
> specified, some defaults are used.
>
> I think we probably also need a modification time attribute, but I haven't
> shown that here.
The modification time is stored in a seperate file which keeps track of
which documents are in the database and is used for database maintenance,
updating, installing, uninstalling, etc. The contents list does not care
about the modification time.
> Now, the user uses sk-install-doc again, but this time they install the
> PDF file.
>
> sk-install-doc -format pdf -db /var/db/sk/master.xml -id MD5 \
> /usr/local/share/doc/app-name/pdf/doc.pdf
>
> A very similar command line. However, because this is another instance of
> the same document, this time the user specifies the ID of the document.
> master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance "url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Finally, the chap that wrote the application also maintains information
> about the document on the web. So the user decides to do the following;
>
> sk-install-doc -format html-split -db /var/db/sk/master.xml -id MD5
> http://www.example.com/app-name/index.html
>
> and master.xml now looks like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="file:/usr/local/share/doc/app-name/pdf/doc.pdf"
> format="pdf" lang="en_US" encoding="ISO_8859-1">
> <docinstance url="http://www.example.com/app-name/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
> </doc>
>
> Now, typically, the end user wouldn't run any of this themselves. It
> is the responsibility of whoever produced the package/rpm/whatever to
> make sure that this happens.
>
> I understand that some people have said that that's too much work to
> expect package maintainers to do. I don't think that's the case. It
> doesn't matter if the original program's author, or the package maintainer
> doesn't want to do it. All it takes is someone to find this functionality
> useful enough to spend the five minutes constructing the sk-* command
> lines to do this, and submit it back to the maintainer. These sorts of
> commands are not going to change a great deal between package releases.
Hopefully we agree that sk should be as simple and robust for the packager
and distributor as possible. This means the least amount of typing with
the smallest possible learning curve, with good fault tolerance.
> [ In fact, I see this as being a differentiator between operating systems.
> SuSE (to pick an example) might not bother generating OMF files for
> applications that don't already provide them. But it might be a point
> of pride for the Debian people that they don't bundle a third party
> application until the person making the .deb file has generated an OMF
> file for it ]
I'd rather Suse and Debian distinguish themselves in other ways than
seeing which one can dedicate more man-hours to making a complicated
document registration system work properly ;) Perhaps they can spend this
time improving sk or some other nifty application.
And unfortunately, it isn't a matter of who does a better job but a matter
of whether sk gets adopted or not. If it is difficult to use, it will not
succeed. If it is adopted, the simpler it is the more packages that will
support it and the more useful it will be to the user.
> OK, so we've told sk about the document, but we haven't pointed to
> any of the OMF data yet. Recall that the OMF data is the same despite
> the different document formats.
Actually they are different OMF files which have different IDENTIFIER's
and FORMAT's.
> The end user (or their package management application) runs
>
> sk-install-omf -id MD5 app-name.omf
>
> Notice how the MD5 checksum (or whatever) is included on the command line,
> so that sk can associate this OMF file with the document that was installed
> earlier.
Having the package determine the id would likely cause problems, but can
safely be omitted since sk can manage the id's itself.
> Now, because we've already installed the document, and sk knows where the
> document has been installed in to, the OMF shouldn't need to contain any
> path data. So there's nothing for sk to need to rewrite. sk can either
> copy app-name.omf to somewhere (/var/db/sk/omf/<md5>.omf), and link to
> it master.xml, or it can copy it wholesale in to master.xml.
See proposal #3 for how this is done. We definitely want to install the
OMF file somewhere other than in the database so that the database can be
rebuilt from original OMF files if it is corrupted.
> The former might look like this;
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf url="file:/var/db/sk/omf/<md5>.omf">
> </doc>
>
> the latter might look like
>
> <doc docid="the MD5 checksum">
> <docinstance url="file:/usr/local/share/doc/app-name/html/index.html"
> format="html-split" lang="en_US" encoding="ISO_8859-1">
>
> <!-- Other <docinstance> elements here -->
>
> <omf>
> <omf:title>The App-name manual</omf:title>
>
> <!-- Other elements here -->
> </omf>
> </doc>
>
> It shouldn't matter to the end user.
>
> The user can now regenerate their HTML list of documents installed on
> the system. They might do something like
>
> sk-generate-contents -format html -db /var/db/master.xml > contents.html
>
> which would generate contents.html, which, in a browser, might look
> something like this;
>
> ==========================================================================
> Category: Unfiled
>
> The App-Name manual
>
> Include meta information from the OMF here (SK ID = MD5)
>
> Formats: Local HTML (split), PDF, Remote HTML (split)
> =========================================================================
This may be useful for people who don't have a help-browser aside from
Netscape. Although Netscape won't know how to display most docs.
> I anticipate that there will be a system wide sk Contents List (like
> /var/db/sk/master.xml in my examples) that will be used to generate
> a system wide HTML (and other formats) Contents List regularly, through
> cron, or something similar.
The database and any exported files like this are updated each time a new
package/doc is installed, uninstalled, or updated. We may have to use
cron if we want to have automatic processing of man and info files since
the install setups for these will not generally call sk.
> Notice that all this works whether the end user is root or not. If the
> documentation has been installed somewhere under $HOME then the user
> can adjust sk-install paths as necessary.
Yes. This is a very nice feature. The help browser can provide a utility
(or one could use an external tool) which allows users to easily create
OMF files to include their own docs and to modify their personal
categories list.
> Install new documentation that doesn't come with OMF
>
> This is very similar to the previous example. However, instead of
> running sk-install-omf, the user has two choices.
>
> 1. Skip the step entirely. In which case, when the contents list is
> generated there won't be a document title, just a filename.
>
> 2. Run one of the helper apps I talked about earlier, that tries to
> parse the document and make a stab at generating the OMF. The
> user can then clean this up hand as necessary, and run sk-install-omf
> by hand (if they're so inclined). They can also submit the OMF
> they've written back to the original author.
>
> Generate TOC and Index
>
> sk-generate-toc -format html -db /var/db/master.xml
>
> sk-generate-index -format html -db /var/db/master.xml
As with the contents list, we could export html. However I think most
browsers will find XML much more useful.
> I'll hand wave over these. Again, they should be cron'able, and probably
> need to support various different output formats.
>
> Removing documentation
>
> If the user wants to remove a single document format, but keep the
> information about the document, they might do
>
> sk-remove-doc -format pdf -id MD5
>
> which would remove the <docinstance> information for the PDF version of
> the document with id == MD5.
>
> If they want to remove a complete document, they do
>
> sk-remove-doc -id MD5
>
> which removes the whole thing. This does not touch the document's files
> (such as /usr/local/share/doc/app-name/*), but it does update the Contents
> List, and if the OMF has been stored in a separate file then it removes
> that as well.
As above, we don't want to specify metadata (eg. the doc format) or the
id's on the command line. The timestamps and existance of OMF files are
sufficient to determine if an entry should be updated or removed.
We could optionally pass the OMF file path or doc file path to a special
uninstall script as was in proposal #1, but this complicates things and is
unnecessary.
[SNIP]
I like a number of these ideas. You clearly spent quite a bit of time on
this and found a number of uses and features we completely missed.
I'm unclear on how and why we should embed contents lists inside each
other. I'm also very scared by the idea of listing metadata on the
command line in addition to inside the OMF metadata file. I think this
creates a lot more work for packagers and risks introducing a lot of
errors and inconsistancies in the metadata without gaining anything. The
obvious exception is the case where an OMF file does not exist, but I
think creating the metadata and installing it should be two seperate
steps.
Dan
|
|
From: Laszlo K. <las...@su...> - 2000-11-22 10:21:36
|
> The current way things are designed is that it will have one contents list > for the system. Of course you can always create another one somewhere > else using the -p flag, but generally they would contain the same > information and only the default one will get updated each time a document > is installed. By specifying which OMF directories you want to include, > these databases could be made to differ. We don't currently have a > mechanism for supporting or specifying multiple OMF directories, but that > would be fairly straightforward and is planned. This will make it very > simple for users to have their own local databases which vary from the > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf > which add to or over-ride the system defaults. We will have to think this through. The current implementation does not comply with the layering of information and configuration on the system (network-wide, system-wide, user). I'll look into this in more detail later and see what I can come up with. I don't know how well Linux complies with this, Solaris does it very well in general. I'll check how they do it with documentation. This is an architectural issue, it is very important to sort it out before we go too deep in development. Laszlo |
|
From: Nik C. <ni...@no...> - 2000-11-23 00:59:35
|
On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > There should be no such thing as a non-relocatable package.
>
> Both relocatable and non-relocatable packages exist, so scrollkeeper
> should support them both in the simplest possible way.
Can you give me an example of non-relocatable package, and it's
justification for being non-relocatable?
> The current way things are designed is that it will have one contents list
> for the system.
I think that's a bad idea.
Also, I should be clear here -- I'm not talking about how the system is
currently designed, or what the code currently does. I'm describing
what I think SK should do, and what the interfaces should be. It's kind
of hard to evaluate the existing functionality at the moment, because I
haven't seen a roadmap that lists explicit design decisions, and things
that will and won't be supported.
> Of course you can always create another one somewhere
> else using the -p flag, but generally they would contain the same
> information and only the default one will get updated each time a document
> is installed. By specifying which OMF directories you want to include,
> these databases could be made to differ. We don't currently have a
> mechanism for supporting or specifying multiple OMF directories, but that
> would be fairly straightforward and is planned. This will make it very
> simple for users to have their own local databases which vary from the
> system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> which add to or over-ride the system defaults.
This now sounds different from design doc #3, which, IIRC, suggested
that the contents list, and the OMF information, would be kept separate.
> It is not clear how and why one would want to use multiple contents lists
> except for individual users and/or hand customization. The problem is
> that somebody has to specify what each database contains. So it works for
> users, because they can hand-tweak the contents. It doesn't work very
> well for system-wide databases, since everything must be specified by the
> packager. So while you could imagine GNOME documents being kept in one
> list and KDE documents being kept in another list, this doesn't really
> achieve anything.
I think it does, as long as a mechanism is provided so that sk can work
out where to put the information if the user doesn't indicate otherwise.
A config file that specifies the default Contents List should suffice.
> The information which would be passed on the command
> line to sk to indicate which list the doc goes in is really metadata which
> belongs in the OMF. OMF and XML provide everything we need to have one
> database and understand the scope of each piece. (eg. whether it is a
> GNOME doc or KDE doc.)
If you support multiple Contents Lists (and you have to, just to support
end users installing docs and not having write access to the system wide
Contents List) then you need to be able to specify which Contents List
to use.
Yes, you could put that information in the OMF, but, as I understand it,
the OMF contains meta information about the document. The location of
the Contents List is not meta information about the document, so should
not be in the OMF.
Also, consider a regular user who occasionally installs applications and
documentation in to $HOME.
It's much easier for them to add a flag to a command line (or, better
still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
than it is for them to edit OMF files before installing them.
> What is the advantage of breaking the main database into pieces?
Flexibility for the future.
> > 2. Maintaining a TOC from the Contents List
> >
> > 3. Maintaining an Index from the Contents List
> >
> > 4. Maintaining a Categories list
>
> Well, this is somewhat debatable and worthy of discussion. Somebody has
> to provide and maintain the categories list. On one side, we can just
> define it as part of OMF/sk and install a static list with sk. Then no
> maintenance is necessary. Or, we may consider having this specified by a
> configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> sys admin could drop in a replacement if they want. For example you could
> chop out all the HOWTO's and man pages if you know newbies who never touch
> a command line will be using your machine.
I don't think SK should mandate the existence of a categories list, or
mandate that certain categories should exist. I expect that some well
defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
"Gnome") will spring up.
However, that's not the same thing as saying
If a categories file exists (in some, as yet undecided format), SK
should have the option of warning the SA that they're about to install
a document with a category that does not currently exist, and asking
whether they want to continue. Naturally, the SA should have the
option of overriding this check with a command line flag.
> > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > application will have the responsibility for handling user queries of the
> > SK TOC and Index files ]
>
> sk should export searching functionality.
How? libsk.so? By having a program that accepts search parameters and
returns XML with the results? Some other mechanism?
> > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > maintain several different TOCs and Indices from several different Contents
> > Lists on the same host (for example, a regular user might want to run sk
> > to maintain their own index and TOC of ~/my-docs).
>
> For a given database, I agree. Eventually, I see the system having one
> database (with one contents list) and possibly each user having one. Is
> this what you have in mind? If you think we want more than one contents
> list in the system's database, please explain why.
Flexibility and efficiency. Suppose you have one big contents list that
contains information about all the documentation that's installed on the
system -- man pages, readmes, faqs, and so on. That's going to be a
very big contents list, particularly when apps like Gnome and KDE get in
on the act.
For example, on a BSD system, the contents of /usr/share/man change once
in a blue moon, as third party manual pages generally live under
/usr/local/share/man. So the admin might want to create a Contents List
for /usr/share/man once, and keep it separate, because the SA knows its
never going to change, so it doesn't need to be included in any
processes SK undertakes to ensure that the Contents List is sync with
the filesystem.
> > What sort of documentation can the Contents List going to point to?
> >
> > 1. Locally installed standard documentation that comes with the system,
> > such as man pages, or Info documents.
>
> Perhaps. I think we probably do want to include this information, using
> automagically generated (and therefore incomplete) OMF metadata as you
> suggest below. For man and info, we would have to update this information
> off of cron unfortunately. So there will be an unavoidable delay between
> when a document is installed and when it is registered with sk.
I don't quite agree. As I say below, I think installing a new document
should be a two step process, at least conceptually (as it could be
carried out by one command).
First, the user tells SK about the documentation.
Second, the user assigns some OMF to the documentation.
This is what I'm getting at with the sk-install-doc and sk-install-omf
command separation.
Consider the man page, for example. If I want to tell SK about all my
section 1 man pages, I'd like to be able to do something like
cd /usr/share/man/man1
sk-install-doc *.1
which would tell SK about all my section 1 manual pages. Actually, what
I'd probably do is something like
sk-install-doc -category "Man Pages:Section 1" *.1
to assign a category to them. At this point, there's no OMF for them,
but SK can synthesise the bare bones from the filename and the category.
Suppose, then, that I want to add some OMF for ls.1. I might do
something like this;
sk-extract-omf ls.1 > ls.omf
vi ls.omf
[edit edit edit]
sk-install-omf ls.1 ls.omf
which would pull out the OMF that SK synthesised for me in the earlier
step (presumably as XML). I can then edit it, and then install the
edited OMF back in to SK's control.
I think it's highly likely that *roff -man (and in particular, *roff
-mdoc) is going to be easy to extract OMF from. So what we might end up
with is a command to extract OMF from -man formatted documents. It
might work like this;
cd /usr/share/man/man1
foreach man in (*.1)
extract-omf -source man $man > `basename $man .1`.omf
end
which would iterate over *.1, extracting the OMF and writing it to
*.omf. With that done, the end user might then be able to do
foreach man in (*.1)
sk-install-doc -doc $man -omf `basename $man .1`.omf
end
which could install the document, and the pregenerated OMF, into SK with
one command.
I hope the design ethos that's coming across from all of this is to have
lots of small tools, each of which does one thing.
> > 2. Locally installed system documentation, installed by the SA team.
>
> SA =? sys admin
Yes.
> > 3. Locally installed documentation, installed by the user.
>
> Yes. Although like much of this discussion, this feature will come in a
> post-0.1 release.
As I say, I'm not thinking in terms of 'release' yet. I'd like to get
some specific functionality nailed down and agreed on.
[ Incidentally, and talking about the implementation -- one of the
things that occured to me last night; sk could have a command
structure like that of cvs, where instead of having a bunch of
sk-foo-bar commands, you have one 'sk' command, that takes options,
and then subcommands, which also take options.
sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
sk install-omf /path/to/omf
and so on. ]
> > I think it is fair to say that, when run, sk will need to construct it's
> > global Contents List by parsing multiple, smaller, contents lists. In much
> > the same way that SGML catalog files can include one another at the moment.
>
> Why? It sounds very nice and wonderful. However, this adds a layer of
> complexity without any obvious benefits.
As above.
Note that I don't mean that SK goes and writes a new, master contents
list file. Just that, in memory, SK is capable of aggregating
information from multiple Contents Lists (which should be accessible by
URL, so that remote Contents Lists are feasible).
> > DD: sk must have a run time option to allow the end user to select which
> > TOC to update.
>
> The databases are stored together. So, this should be specified at the
> same time as the contents list part of the database. I don't think I want
> my contents list coming from one database and my toc from another.
>
> (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> TOC.)
It's my understanding that there's a 1:1 mapping between a Contents List
and a ToC. That is, you can't have a ToC that includes information
about documents that aren't in the Contents List.
So if you have multiple Contents Lists, you need to have multiple ToCs
(and Indexes), and a way of saying that >this< Contents List was used to
produce >This< ToC.
Or are you saying that the ToC should be stored within the Contents List?
> > DD: sk must have a run time option to allow the end user to select which
> > Index to update.
>
> (as above)
Ditto :-)
> > DD: One of the sk commands must be to generate a usable Contents List
> > (which really means {X}HTML version, to start with) given a Contents
> > List which is in sk internal format.
>
> Everything is XML right now, so this is not an issue. If we decide
> another internal format is better we should still export XML to the
> browser. As for HTML, this is possible and may be a nice interface for
> people who want to create a very minimalistic web browser.
All the world is not yet XML, and not everyone runs KDE or Gnome. Being
able to convert the ToCs and Indices to other formats is vital.
Fortunately, since the ToC and Indices are XML, this should be easy,
using something like XSLT, so we can probably just consider this
functionality that we get for free.
> > DD: It is possible for a document to change, but for the OMF information
> > to be neglected. sk should maintain a modification time stamp for
> > each document and its corresponding OMF, to allow the end user to
> > determine when that might have happened, and to correct for it.
>
> For the current version of sk, which does not yet allow for remote
> documents, only the OMF file's timestamp is watched. This is because if
> somebody edits the doc and not the OMF, then it is their fault for not
> updating the OMF, not sk's. In reality whenever you install a new version
> of a package, even if the OMF is not updated (as it should), it still
> updates the old version so it gets a new timestamp and the new doc gets
> registered despite any negligence on the part of the maintainer to update
> the OMF (or the doc for that matter;).
>
> This mainly becomes an issue for:
> 1) remote docs
> 2) docs which users write and modify themselves.
>
> So we may indeed need to start tracking the doc timestamps when we support
> these features. We have to be careful about constantly checking the
> timestamps on a bunch of docs which are spread around the internet
> though. There will be many cases where this task cannot complete, or else
> takes an unacceptable amount of time.
This should be configurable behaviour, something like;
sk --check-remote-timestamps --remote-timeout=30
Actually, this implies another DD
DD: sk must track the validity of the OMF in relation to it's document.
OMF might be:
valid This OMF was installed with the document, and neither
the document or the OMF's timestamps have changed.
invalid The document and OMF have different timestamps
unknown The document and OMF have the same timestamps, but
the most recent attempt to confirm this (particularly
for remote documents) failed.
sk should be able to report on documents who have OMF which is not
valid, so the SA can investigate further.
> > DD: One document, multiple output formats. The user might have installed
> > the same document three times, once as HTML, once as PDF, and once as
> > plain text. The three installations are the same document, and have
> > the same OMF associated with them.
>
> No. These definitely do not have the same OMF, unless they are identical
> copies. The OMF file specifies the document format in FORMAT.DTD and
> FORMAT.MIME. The docs also have different IDENTIFIER's.
>
> (We may decide to expand this a little bit. FORMAT.STYLESHEET may be
> nice, as it would allow the doc authors to dictate the appearance of their
> doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> icon, etc.)
>
> > sk should treat this as one document with three formats, rather than
> > three separate documents.
>
> If you look in the ldp-meta archives from several months ago, I proposed
> we add a document id to the OMF. This would allow us to identify if two
> documents are "The-Same". I call two documents "The-Same" if they are the
> same document only different versions, languages, formats, etc. If two
> documents have the same title, they are not necessarily
> "The-Same". Likewise, if I change the title of my document at some point
> in time, I can have two docs which are "The-Same" but not having the same
> title.
>
> Obviously this sort of information will be very nice to have. You can do
> things like "Get me this document in language xxx." or "Show me the latest
> version of this document on the Net." or "Show me the previous version of
> this document from the Net."
>
> > [ This is probably going to be the most contentious piece of this
> > writeup, and it might change. In particular, it also suggests that
> > translations of a document should be treated as the same document,
> > just in a different language, and I'm not sure that's a good idea.
> > Then again, it might be. ]
>
> I guess we were thinking along similar lines :)
Do you agree that making the distinction between a document, and one or
more instances of that document (different formats, different languages)
is worthwhile.
> > There are a number of tasks that the sk end user is going to want to tell
> > sk to do. They are;
> >
> > Install new documentation that comes bundled with OMF
> >
> > The user has just downloaded and installed a package that comes with one
> > or more bundled documents, and the package maintainer has thoughtfully
> > provided an OMF file for each document.
> >
> > Each document has been provided in HTML and PDF format.
> >
> > Suppose the user installs the HTML documentation first, and installs it
> > it into /usr/local/share/doc/app-name/html/, where normal file is called
> > index.html.
> >
> > [ Everywhere I say "the user runs" I really mean "The package maintainer
> > does this by means of a post-install script that they've already written.
> > It is assumed that this post-install script knows the directories that
> > documentation has been installed in, by virtue of being part of the
> > package that did the installation in the first place. ]
> >
> > The user runs
> >
> > sk-install-doc -format html-split -db /var/db/sk/master.xml \
> > /usr/local/share/doc/app-name/html/index.html
>
> Why do we specify the format on the command line? The FORMAT is specified
> in the OMF already. The packager should just install and register the
> metadata instead of putting it on the command line.
Because at this point, all we've done is install a document with no OMF.
SK has to synthesise some OMF, based on any additional information we
give it.
This separation is necessary to support documents that have no separate
OMF. Adding "-format" to the command line is probably going to be
easier for an end user than having to create a separate file to return
to.
> > 'sk-install-doc' registers the document. The two (optional) arguments
> > here are
> >
> > -format The format of the installed documentation. Might be
> > html-split - Bunch of small HTML files
> > html - One big HTML file
> > ps - postscript
> > pdf - PDF
> > ... - extend as necessary (rtf, txt, pdb, ...)
> >
> > If not specified, sk can try and guess what it is.
>
> I don't think we really want this. One could make an argument that if no
> OMF fie exists we could have a command like 'scrollkeeper-build-omf' which
> has a bunch of arguments for each element in the OMF. However I'm not
> sure if we want to merge all this into the same command that registers the
> document with sk.
Six of one, half a dozen of another. Certainly, a sequence that went
something like
sk-build-omf -format html ... > doc.omf
sk-install-doc <options here> index.html
sk-install-omf <options here> doc.omf
would also work.
> > -db The Contents List to update. Uses a default if not specified.
>
> Here I assume (hope?;) that you mean the database directory and not the
> contents list. If so, we already do this with the "-p" flag.
Right.
> > [ Note: This also needs a category option, so that the end user can
> > specify which categories the document goes in to. By default,
> > it goes in to "unfiled". ]
>
> Here we go, implementing the OMF on the command-line ;)
> We should put all this metadata in the OMF metadata file.
I'd like to see a mechanism to specify it on the command line (apart
from anything else, it's easier to script). sk-build-omf, as you
suggest above, seems like a good compromise.
> > This command does not physically copy the file in to place.
> >
> > sk-install-doc should generate a unique identifier for this document
> > (MD5 hash of the filename?), and display this to the user. We probably
> > also need a standalone utility (sk-generate-id?) that, given a path,
> > generates an ID.
>
> Right now we just number them, which guarantees a unique identifier.
Two problems with this:
1. Probably won't scale efficiently as the Contents List gets larger.
2. Makes it impossible to merge two separate Contents Lists, from two
or more sites (say you're on a univesity campus, where the biology
department and the chemistry department both make available their
list of documentation in two different Scroll Keeper Contents Lists).
Your document browser is going to need a mechanism to uniquely
identify each document. If there's a strong probability that the
document IDs are not unique then the browsers are each going to
have to reinvent their own mechanism for ensuring that the
documents are unique.
The documents URL might be appropriate, but it's a bit unwieldy.
You really need a mechanism that can (as near as possible) guarantee
globally unique document IDs.
> > As you can see, not a lot of information in there so far. "lang" and
> > "encoding" could also be specified on the command line, and if not
> > specified, some defaults are used.
> >
> > I think we probably also need a modification time attribute, but I haven't
> > shown that here.
>
> The modification time is stored in a seperate file which keeps track of
> which documents are in the database and is used for database maintenance,
> updating, installing, uninstalling, etc. The contents list does not care
> about the modification time.
It probably should. Apart from the issues I outline above, this then
also lets you do things like "Show me the documents in order, most
recently modified first".
> Hopefully we agree that sk should be as simple and robust for the packager
> and distributor as possible. This means the least amount of typing with
> the smallest possible learning curve, with good fault tolerance.
Yes. Having said that, we shouldn't be afraid to aggregate certain
functionality in to one tool.
For example, cp(1), mv(1), chmod(1), and chown(1) all do one task, and
do it well. That doesn't stop install(1) from being useful in its own
right.
> > OK, so we've told sk about the document, but we haven't pointed to
> > any of the OMF data yet. Recall that the OMF data is the same despite
> > the different document formats.
>
> Actually they are different OMF files which have different IDENTIFIER's
> and FORMAT's.
OK. I'm going to poke around OMF in more detail, and think about this
issue some more.
> > ==========================================================================
> > Category: Unfiled
> >
> > The App-Name manual
> >
> > Include meta information from the OMF here (SK ID = MD5)
> >
> > Formats: Local HTML (split), PDF, Remote HTML (split)
> > =========================================================================
>
> This may be useful for people who don't have a help-browser aside from
> Netscape. Although Netscape won't know how to display most docs.
True. However, we should allow the Index builder flexibilit in deciding
what documents are included in the Index. This may include producing a
special Index that only contains documents available in HTML, plain
text, and PDF (for example).
> > Generate TOC and Index
> >
> > sk-generate-toc -format html -db /var/db/master.xml
> >
> > sk-generate-index -format html -db /var/db/master.xml
>
> As with the contents list, we could export html. However I think most
> browsers will find XML much more useful.
As I say, this is a simple XSLT transformation. As long as the source
format is XML this should be relatively simple.
> I'm unclear on how and why we should embed contents lists inside each
> other.
Hopefully I've answered that.
> I'm also very scared by the idea of listing metadata on the
> command line in addition to inside the OMF metadata file. I think this
> creates a lot more work for packagers and risks introducing a lot of
> errors and inconsistancies in the metadata without gaining anything. The
> obvious exception is the case where an OMF file does not exist, but I
> think creating the metadata and installing it should be two seperate
> steps.
I'm only considering it for the cases where there is no OMF, and you
need to bootstrap it in to SK.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Dan M. <d-...@uc...> - 2000-11-23 03:55:48
|
On Thu, 23 Nov 2000, Nik Clayton wrote:
> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> >
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
>
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?
On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
gives me 506 packages which are non-relocateable, the last of which is
xbill:)
I think Red Hat would be better qualified at explaining this decision.
> > The current way things are designed is that it will have one contents list
> > for the system.
>
> I think that's a bad idea.
>
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does. I'm describing
> what I think SK should do, and what the interfaces should be. It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
>
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed. By specifying which OMF directories you want to include,
> > these databases could be made to differ. We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned. This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
>
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.
I must have been unclear above. We definitely want to keep all of the
original metadata in OMF files and seperate from the database. These OMF
files should never be edited after they are initially installed (and
possibly initially fixed to have the right IDENTIFIER during the
post-install script, for relocatable packages). This way we are
guaranteed to have an authoritative copy preserved in case the databases
are corrupted and to make upgrades simpler.
What I was trying to say above is that you can specify where the database
lives with a -p <path> flag. If I run scrollkeeper-update -p /tmp/junk/
and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
all of the OMF files in the directory(ies) it is configured to read, and
then dumps a brand new database(ie. contents list, toc, index, ...) under
/tmp/junk/. This database won't generally be any different from the one in
its normal location (/var/lib/scrollkeeper/) unless it is running under a
different configuration. This would happen for a user who has his own
configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
should be added to the database (typically under $HOME/.scrollkeeper/db/
perhaps).
> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization. The problem is
> > that somebody has to specify what each database contains. So it works for
> > users, because they can hand-tweak the contents. It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager. So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.
>
> I think it does, as long as a mechanism is provided so that sk can work
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
>
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF. OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
>
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.
We definitely want to support multiple databases, each of which has a
Contents List.
I am still uncertain if we want inheritance between contents lists though.
In many cases users will not want inheritance, because they lose any
control over what they inherit. If instead they generate their own
database, they have complete control over all its parts, including those
parts which they would have otherwise inherited. One way they could do
this is to over-ride one or more OMF files by keeping local
copies. Another way is to have a local categories file which differs from
the system's. This approach gives the user complete control over their
database (including their contents lists) which would be much more
difficult and awkward to achieve by inheriting.
Now we can come up with instances where inheritance is useful. If a user
wanted all the man pages in their help system, they might be better served
by inheriting the system man page database. They would be limited though,
in that they could not specify which sections of the man pages they want.
This would be solved by having a database for each section of man
pages. Then these sections could be inherited by users which would be
much faster than regenerating them for each user.
I think we really need to be very clear on how this will work and how it
will be used. Would all databases be generated with the same categories
list? If not, how do you handle merging them? Presumably there is a
"default" categories list which gets enforced on the final contents list.
I think performance measurements would be useful here. The example above
of using inherited databases with man pages has an advantage because
updating the database is faster. It may save disk space, but this is
probably negligable. It has the downside of slowing down run-time
performance and making things substantially more complex and fragile.
> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document. The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.
If I understood you, the "system" (ie nothing under /home) would have
multiple databases. So when documentation is registered, it must be
determined which database to register it in. Since this must be passed to
the registration script, it is determined by the packager based on
information about the content of the docs. Thus, it is based on document
metadata. (eg. this is a man page so it goes in the man page database,
this one is a KDE doc so it goes in the KDE database, ...) This type of
metadata is stored in the OMF file already.
What various databases do you envision?
[SNIP]
> > What is the advantage of breaking the main database into pieces?
>
> Flexibility for the future.
>
> > > 2. Maintaining a TOC from the Contents List
> > >
> > > 3. Maintaining an Index from the Contents List
> > >
> > > 4. Maintaining a Categories list
> >
> > Well, this is somewhat debatable and worthy of discussion. Somebody has
> > to provide and maintain the categories list. On one side, we can just
> > define it as part of OMF/sk and install a static list with sk. Then no
> > maintenance is necessary. Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> > sys admin could drop in a replacement if they want. For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
>
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist. I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.
We could provide an option to not use categories lists. We definitely
should provide the ability to use them however, and we should specify a
suggested list.
> However, that's not the same thing as saying
>
> If a categories file exists (in some, as yet undecided format), SK
> should have the option of warning the SA that they're about to install
> a document with a category that does not currently exist, and asking
> whether they want to continue. Naturally, the SA should have the
> option of overriding this check with a command line flag.
We could provide a warning and even an interactive mode as an option.
> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > application will have the responsibility for handling user queries of the
> > > SK TOC and Index files ]
> >
> > sk should export searching functionality.
>
> How? libsk.so? By having a program that accepts search parameters and
> returns XML with the results? Some other mechanism?
We definitely want libsk.so. I'm not sure if we would want a command-line
program for this.
> > > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > > maintain several different TOCs and Indices from several different Contents
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> >
> > For a given database, I agree. Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one. Is
> > this what you have in mind? If you think we want more than one contents
> > list in the system's database, please explain why.
>
> Flexibility and efficiency. Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on. That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.
It will be big. Hopefully we can start running some tests on performance
some time soon to see how big of an issue this becomes. It is not clear
that merging two medium sized lists into a big list is any better than
starting with a big list. You still have a big list in the end, only you
save the part where you have to process and store two medium lists all the
time.
> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man. So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.
This would speed things up a bit when one installs a new OMF file if man
pages were treated the same as other documents. However processing of man
pages are done seperate from other applications since the OMF files for
most (all?) man pages are generated and can be stored in a special
location. This way the man pages are updated with cron each night (which
must also make sure to create OMF metadata for each man page) and the
normal docs are registered whenever an application installs. Because they
are processed at different times they do not slow each other down.
Aside from rebuilding the database in cases where it becomes corrupt, I'd
like this to be a zero maintenance system that works out of the box. I
don't think we should focus on unusual things a SA may want to
do. Instead we should make it work well and in a robust manner so the SA
won't be tempted to try to "fix" it.
> > > What sort of documentation can the Contents List going to point to?
> > >
> > > 1. Locally installed standard documentation that comes with the system,
> > > such as man pages, or Info documents.
> >
> > Perhaps. I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below. For man and info, we would have to update this information
> > off of cron unfortunately. So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
>
> I don't quite agree. As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
>
> First, the user tells SK about the documentation.
>
> Second, the user assigns some OMF to the documentation.
>
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.
Why do we need the former step? The OMF data includes the IDENTIFIER,
which makes the first step unnecessary.
[SNIP]
> Note that I don't mean that SK goes and writes a new, master contents
> list file. Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).
If we import a contents list it could only have documents on the Net on it
or the docs would generally be unreachable. However this could be a very
useful feature. You could imagine a given project publishing a database
and docs on the Net and then importing them into your local
system. Without the ability to inherit contents list one would have to
install the metadata locally. This would be possible and could even be
automated. We would need to consider this carefully to determine which
solution is simpler and more robust.
> > > DD: sk must have a run time option to allow the end user to select which
> > > TOC to update.
> >
> > The databases are stored together. So, this should be specified at the
> > same time as the contents list part of the database. I don't think I want
> > my contents list coming from one database and my toc from another.
> >
> > (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> > TOC.)
>
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC. That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.
Yes.
> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.
Not if only one contents list/toc pair live in each database directory.
> Or are you saying that the ToC should be stored within the Contents List?
No.
[SNIP]
> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.
The only way scrollkeeper knows that two docs of different
versions/formats/languages are otherwise the same is using the extra OMF
element (which doesn't yet exist) which assigns them an ISBN-type code
which is the same. Since they have different IDENTIFIERS (and other
metadata), sk knows they are different documents.
[SNIP]
> > Why do we specify the format on the command line? The FORMAT is specified
> > in the OMF already. The packager should just install and register the
> > metadata instead of putting it on the command line.
>
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we
> give it.
>
> This separation is necessary to support documents that have no separate
> OMF. Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.
Perhaps we agree on this after all and are just talking past each
other. So for docs with OMF we have two steps:
1) update the IDENTIFIER to point to the right URL
2) register the OMF file with sk
Really, #1 is only necessary for local docs (as opposed to remote
ones) and for relocatable packages.
For docs without OMF, we do two steps:
1) generate the best OMF file we can given the doc
2) register the OMF file with sk
> > Here I assume (hope?;) that you mean the database directory and not the
> > contents list. If so, we already do this with the "-p" flag.
>
> Right.
So we would have multiple databases, each with exactly one Contents List,
TOC directory, etc., right?
> I'd like to see a mechanism to specify it on the command line (apart
> from anything else, it's easier to script). sk-build-omf, as you
> suggest above, seems like a good compromise.
:)
> > Right now we just number them, which guarantees a unique identifier.
>
> Two problems with this:
>
> 1. Probably won't scale efficiently as the Contents List gets larger.
>
> 2. Makes it impossible to merge two separate Contents Lists, from two
> or more sites (say you're on a univesity campus, where the biology
> department and the chemistry department both make available their
> list of documentation in two different Scroll Keeper Contents Lists).
When sk merges seperate databases, it can easily mark them so that it
knows where the pieces came. One way to do this would be to append a few
digit code to the beginning of each id which describes which database it
came from.
> Your document browser is going to need a mechanism to uniquely
> identify each document. If there's a strong probability that the
> document IDs are not unique then the browsers are each going to
> have to reinvent their own mechanism for ensuring that the
> documents are unique.
>
> The documents URL might be appropriate, but it's a bit unwieldy.
The document URL is a very good unique identifier, and it is already being
passed to the help browser.
> You really need a mechanism that can (as near as possible) guarantee
> globally unique document IDs.
I'm pretty sure that only one document lives at a given URL ;)
> > > As you can see, not a lot of information in there so far. "lang" and
> > > "encoding" could also be specified on the command line, and if not
> > > specified, some defaults are used.
> > >
> > > I think we probably also need a modification time attribute, but I haven't
> > > shown that here.
> >
> > The modification time is stored in a seperate file which keeps track of
> > which documents are in the database and is used for database maintenance,
> > updating, installing, uninstalling, etc. The contents list does not care
> > about the modification time.
>
> It probably should. Apart from the issues I outline above, this then
> also lets you do things like "Show me the documents in order, most
> recently modified first".
This sort of filtering should be handled by sk, which knows the timestamps
even though they are not part of the contents list. (The contents list is
there to sort the documents according to a given set of instructions - the
categories file. It is not the primary source for the metadata or
metametadata. We have a seperate metametadata file.)
> > > ==========================================================================
> > > Category: Unfiled
> > >
> > > The App-Name manual
> > >
> > > Include meta information from the OMF here (SK ID = MD5)
> > >
> > > Formats: Local HTML (split), PDF, Remote HTML (split)
> > > =========================================================================
> >
> > This may be useful for people who don't have a help-browser aside from
> > Netscape. Although Netscape won't know how to display most docs.
>
> True. However, we should allow the Index builder flexibilit in deciding
> what documents are included in the Index. This may include producing a
> special Index that only contains documents available in HTML, plain
> text, and PDF (for example).
What do you mean by "index"? The definition I gave for "index" in prop#3
is what you find in the back of a book - an alphabetical list of concepts
and links to where they are discussed in the given work. My impression is
that you mean the Contents List here. Is that right?
I agree that we will want searching/filtering features in sk that allow
people to control its behavior as you suggest.
Dan
|
|
From: Laszlo K. <las...@su...> - 2000-11-23 10:02:01
|
Guys,
You have to cut back on the email size, otherwise nobody is going to
contribute to this discussion.
Laszlo
>
> On Thu, 23 Nov 2000, Nik Clayton wrote:
>
> > On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > > There should be no such thing as a non-relocatable package.
> > >
> > > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > > should support them both in the simplest possible way.
> >
> > Can you give me an example of non-relocatable package, and it's
> > justification for being non-relocatable?
>
> On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> gives me 506 packages which are non-relocateable, the last of which is
> xbill:)
>
> I think Red Hat would be better qualified at explaining this decision.
>
> > > The current way things are designed is that it will have one contents list
> > > for the system.
> >
> > I think that's a bad idea.
> >
> > Also, I should be clear here -- I'm not talking about how the system is
> > currently designed, or what the code currently does. I'm describing
> > what I think SK should do, and what the interfaces should be. It's kind
> > of hard to evaluate the existing functionality at the moment, because I
> > haven't seen a roadmap that lists explicit design decisions, and things
> > that will and won't be supported.
> >
> > > Of course you can always create another one somewhere
> > > else using the -p flag, but generally they would contain the same
> > > information and only the default one will get updated each time a document
> > > is installed. By specifying which OMF directories you want to include,
> > > these databases could be made to differ. We don't currently have a
> > > mechanism for supporting or specifying multiple OMF directories, but that
> > > would be fairly straightforward and is planned. This will make it very
> > > simple for users to have their own local databases which vary from the
> > > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > > which add to or over-ride the system defaults.
> >
> > This now sounds different from design doc #3, which, IIRC, suggested
> > that the contents list, and the OMF information, would be kept separate.
>
> I must have been unclear above. We definitely want to keep all of the
> original metadata in OMF files and seperate from the database. These OMF
> files should never be edited after they are initially installed (and
> possibly initially fixed to have the right IDENTIFIER during the
> post-install script, for relocatable packages). This way we are
> guaranteed to have an authoritative copy preserved in case the databases
> are corrupted and to make upgrades simpler.
>
> What I was trying to say above is that you can specify where the database
> lives with a -p <path> flag. If I run scrollkeeper-update -p /tmp/junk/
> and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
> all of the OMF files in the directory(ies) it is configured to read, and
> then dumps a brand new database(ie. contents list, toc, index, ...) under
> /tmp/junk/. This database won't generally be any different from the one in
> its normal location (/var/lib/scrollkeeper/) unless it is running under a
> different configuration. This would happen for a user who has his own
> configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
> should be added to the database (typically under $HOME/.scrollkeeper/db/
> perhaps).
>
> > > It is not clear how and why one would want to use multiple contents lists
> > > except for individual users and/or hand customization. The problem is
> > > that somebody has to specify what each database contains. So it works for
> > > users, because they can hand-tweak the contents. It doesn't work very
> > > well for system-wide databases, since everything must be specified by the
> > > packager. So while you could imagine GNOME documents being kept in one
> > > list and KDE documents being kept in another list, this doesn't really
> > > achieve anything.
> >
> > I think it does, as long as a mechanism is provided so that sk can work
> > out where to put the information if the user doesn't indicate otherwise.
> > A config file that specifies the default Contents List should suffice.
> >
> > > The information which would be passed on the command
> > > line to sk to indicate which list the doc goes in is really metadata which
> > > belongs in the OMF. OMF and XML provide everything we need to have one
> > > database and understand the scope of each piece. (eg. whether it is a
> > > GNOME doc or KDE doc.)
> >
> > If you support multiple Contents Lists (and you have to, just to support
> > end users installing docs and not having write access to the system wide
> > Contents List) then you need to be able to specify which Contents List
> > to use.
>
> We definitely want to support multiple databases, each of which has a
> Contents List.
>
> I am still uncertain if we want inheritance between contents lists though.
>
> In many cases users will not want inheritance, because they lose any
> control over what they inherit. If instead they generate their own
> database, they have complete control over all its parts, including those
> parts which they would have otherwise inherited. One way they could do
> this is to over-ride one or more OMF files by keeping local
> copies. Another way is to have a local categories file which differs from
> the system's. This approach gives the user complete control over their
> database (including their contents lists) which would be much more
> difficult and awkward to achieve by inheriting.
>
> Now we can come up with instances where inheritance is useful. If a user
> wanted all the man pages in their help system, they might be better served
> by inheriting the system man page database. They would be limited though,
> in that they could not specify which sections of the man pages they want.
> This would be solved by having a database for each section of man
> pages. Then these sections could be inherited by users which would be
> much faster than regenerating them for each user.
>
> I think we really need to be very clear on how this will work and how it
> will be used. Would all databases be generated with the same categories
> list? If not, how do you handle merging them? Presumably there is a
> "default" categories list which gets enforced on the final contents list.
>
> I think performance measurements would be useful here. The example above
> of using inherited databases with man pages has an advantage because
> updating the database is faster. It may save disk space, but this is
> probably negligable. It has the downside of slowing down run-time
> performance and making things substantially more complex and fragile.
>
> > Yes, you could put that information in the OMF, but, as I understand it,
> > the OMF contains meta information about the document. The location of
> > the Contents List is not meta information about the document, so should
> > not be in the OMF.
>
> If I understood you, the "system" (ie nothing under /home) would have
> multiple databases. So when documentation is registered, it must be
> determined which database to register it in. Since this must be passed to
> the registration script, it is determined by the packager based on
> information about the content of the docs. Thus, it is based on document
> metadata. (eg. this is a man page so it goes in the man page database,
> this one is a KDE doc so it goes in the KDE database, ...) This type of
> metadata is stored in the OMF file already.
>
> What various databases do you envision?
>
> [SNIP]
>
> > > What is the advantage of breaking the main database into pieces?
> >
> > Flexibility for the future.
> >
> > > > 2. Maintaining a TOC from the Contents List
> > > >
> > > > 3. Maintaining an Index from the Contents List
> > > >
> > > > 4. Maintaining a Categories list
> > >
> > > Well, this is somewhat debatable and worthy of discussion. Somebody has
> > > to provide and maintain the categories list. On one side, we can just
> > > define it as part of OMF/sk and install a static list with sk. Then no
> > > maintenance is necessary. Or, we may consider having this specified by a
> > > configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> > > sys admin could drop in a replacement if they want. For example you could
> > > chop out all the HOWTO's and man pages if you know newbies who never touch
> > > a command line will be using your machine.
> >
> > I don't think SK should mandate the existence of a categories list, or
> > mandate that certain categories should exist. I expect that some well
> > defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> > "Gnome") will spring up.
>
> We could provide an option to not use categories lists. We definitely
> should provide the ability to use them however, and we should specify a
> suggested list.
>
> > However, that's not the same thing as saying
> >
> > If a categories file exists (in some, as yet undecided format), SK
> > should have the option of warning the SA that they're about to install
> > a document with a category that does not currently exist, and asking
> > whether they want to continue. Naturally, the SA should have the
> > option of overriding this check with a command line flag.
>
> We could provide a warning and even an interactive mode as an option.
>
> > > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > > application will have the responsibility for handling user queries of the
> > > > SK TOC and Index files ]
> > >
> > > sk should export searching functionality.
> >
> > How? libsk.so? By having a program that accepts search parameters and
> > returns XML with the results? Some other mechanism?
>
> We definitely want libsk.so. I'm not sure if we would want a command-line
> program for this.
>
> > > > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > > > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > > > maintain several different TOCs and Indices from several different Contents
> > > > Lists on the same host (for example, a regular user might want to run sk
> > > > to maintain their own index and TOC of ~/my-docs).
> > >
> > > For a given database, I agree. Eventually, I see the system having one
> > > database (with one contents list) and possibly each user having one. Is
> > > this what you have in mind? If you think we want more than one contents
> > > list in the system's database, please explain why.
> >
> > Flexibility and efficiency. Suppose you have one big contents list that
> > contains information about all the documentation that's installed on the
> > system -- man pages, readmes, faqs, and so on. That's going to be a
> > very big contents list, particularly when apps like Gnome and KDE get in
> > on the act.
>
> It will be big. Hopefully we can start running some tests on performance
> some time soon to see how big of an issue this becomes. It is not clear
> that merging two medium sized lists into a big list is any better than
> starting with a big list. You still have a big list in the end, only you
> save the part where you have to process and store two medium lists all the
> time.
>
> > For example, on a BSD system, the contents of /usr/share/man change once
> > in a blue moon, as third party manual pages generally live under
> > /usr/local/share/man. So the admin might want to create a Contents List
> > for /usr/share/man once, and keep it separate, because the SA knows its
> > never going to change, so it doesn't need to be included in any
> > processes SK undertakes to ensure that the Contents List is sync with
> > the filesystem.
>
> This would speed things up a bit when one installs a new OMF file if man
> pages were treated the same as other documents. However processing of man
> pages are done seperate from other applications since the OMF files for
> most (all?) man pages are generated and can be stored in a special
> location. This way the man pages are updated with cron each night (which
> must also make sure to create OMF metadata for each man page) and the
> normal docs are registered whenever an application installs. Because they
> are processed at different times they do not slow each other down.
>
> Aside from rebuilding the database in cases where it becomes corrupt, I'd
> like this to be a zero maintenance system that works out of the box. I
> don't think we should focus on unusual things a SA may want to
> do. Instead we should make it work well and in a robust manner so the SA
> won't be tempted to try to "fix" it.
>
> > > > What sort of documentation can the Contents List going to point to?
> > > >
> > > > 1. Locally installed standard documentation that comes with the system,
> > > > such as man pages, or Info documents.
> > >
> > > Perhaps. I think we probably do want to include this information, using
> > > automagically generated (and therefore incomplete) OMF metadata as you
> > > suggest below. For man and info, we would have to update this information
> > > off of cron unfortunately. So there will be an unavoidable delay between
> > > when a document is installed and when it is registered with sk.
> >
> > I don't quite agree. As I say below, I think installing a new document
> > should be a two step process, at least conceptually (as it could be
> > carried out by one command).
> >
> > First, the user tells SK about the documentation.
> >
> > Second, the user assigns some OMF to the documentation.
> >
> > This is what I'm getting at with the sk-install-doc and sk-install-omf
> > command separation.
>
> Why do we need the former step? The OMF data includes the IDENTIFIER,
> which makes the first step unnecessary.
>
> [SNIP]
>
> > Note that I don't mean that SK goes and writes a new, master contents
> > list file. Just that, in memory, SK is capable of aggregating
> > information from multiple Contents Lists (which should be accessible by
> > URL, so that remote Contents Lists are feasible).
>
> If we import a contents list it could only have documents on the Net on it
> or the docs would generally be unreachable. However this could be a very
> useful feature. You could imagine a given project publishing a database
> and docs on the Net and then importing them into your local
> system. Without the ability to inherit contents list one would have to
> install the metadata locally. This would be possible and could even be
> automated. We would need to consider this carefully to determine which
> solution is simpler and more robust.
>
> > > > DD: sk must have a run time option to allow the end user to select which
> > > > TOC to update.
> > >
> > > The databases are stored together. So, this should be specified at the
> > > same time as the contents list part of the database. I don't think I want
> > > my contents list coming from one database and my toc from another.
> > >
> > > (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> > > TOC.)
> >
> > It's my understanding that there's a 1:1 mapping between a Contents List
> > and a ToC. That is, you can't have a ToC that includes information
> > about documents that aren't in the Contents List.
>
> Yes.
>
> > So if you have multiple Contents Lists, you need to have multiple ToCs
> > (and Indexes), and a way of saying that >this< Contents List was used to
> > produce >This< ToC.
>
> Not if only one contents list/toc pair live in each database directory.
>
> > Or are you saying that the ToC should be stored within the Contents List?
>
> No.
>
> [SNIP]
>
> > Do you agree that making the distinction between a document, and one or
> > more instances of that document (different formats, different languages)
> > is worthwhile.
>
> The only way scrollkeeper knows that two docs of different
> versions/formats/languages are otherwise the same is using the extra OMF
> element (which doesn't yet exist) which assigns them an ISBN-type code
> which is the same. Since they have different IDENTIFIERS (and other
> metadata), sk knows they are different documents.
>
> [SNIP]
>
> > > Why do we specify the format on the command line? The FORMAT is specified
> > > in the OMF already. The packager should just install and register the
> > > metadata instead of putting it on the command line.
> >
> > Because at this point, all we've done is install a document with no OMF.
> > SK has to synthesise some OMF, based on any additional information we
> > give it.
> >
> > This separation is necessary to support documents that have no separate
> > OMF. Adding "-format" to the command line is probably going to be
> > easier for an end user than having to create a separate file to return
> > to.
>
> Perhaps we agree on this after all and are just talking past each
> other. So for docs with OMF we have two steps:
> 1) update the IDENTIFIER to point to the right URL
> 2) register the OMF file with sk
>
> Really, #1 is only necessary for local docs (as opposed to remote
> ones) and for relocatable packages.
>
> For docs without OMF, we do two steps:
> 1) generate the best OMF file we can given the doc
> 2) register the OMF file with sk
>
> > > Here I assume (hope?;) that you mean the database directory and not the
> > > contents list. If so, we already do this with the "-p" flag.
> >
> > Right.
>
> So we would have multiple databases, each with exactly one Contents List,
> TOC directory, etc., right?
>
> > I'd like to see a mechanism to specify it on the command line (apart
> > from anything else, it's easier to script). sk-build-omf, as you
> > suggest above, seems like a good compromise.
>
> :)
>
> > > Right now we just number them, which guarantees a unique identifier.
> >
> > Two problems with this:
> >
> > 1. Probably won't scale efficiently as the Contents List gets larger.
> >
> > 2. Makes it impossible to merge two separate Contents Lists, from two
> > or more sites (say you're on a univesity campus, where the biology
> > department and the chemistry department both make available their
> > list of documentation in two different Scroll Keeper Contents Lists).
>
> When sk merges seperate databases, it can easily mark them so that it
> knows where the pieces came. One way to do this would be to append a few
> digit code to the beginning of each id which describes which database it
> came from.
>
> > Your document browser is going to need a mechanism to uniquely
> > identify each document. If there's a strong probability that the
> > document IDs are not unique then the browsers are each going to
> > have to reinvent their own mechanism for ensuring that the
> > documents are unique.
> >
> > The documents URL might be appropriate, but it's a bit unwieldy.
>
> The document URL is a very good unique identifier, and it is already being
> passed to the help browser.
>
> > You really need a mechanism that can (as near as possible) guarantee
> > globally unique document IDs.
>
> I'm pretty sure that only one document lives at a given URL ;)
>
> > > > As you can see, not a lot of information in there so far. "lang" and
> > > > "encoding" could also be specified on the command line, and if not
> > > > specified, some defaults are used.
> > > >
> > > > I think we probably also need a modification time attribute, but I haven't
> > > > shown that here.
> > >
> > > The modification time is stored in a seperate file which keeps track of
> > > which documents are in the database and is used for database maintenance,
> > > updating, installing, uninstalling, etc. The contents list does not care
> > > about the modification time.
> >
> > It probably should. Apart from the issues I outline above, this then
> > also lets you do things like "Show me the documents in order, most
> > recently modified first".
>
> This sort of filtering should be handled by sk, which knows the timestamps
> even though they are not part of the contents list. (The contents list is
> there to sort the documents according to a given set of instructions - the
> categories file. It is not the primary source for the metadata or
> metametadata. We have a seperate metametadata file.)
>
> > > > ==========================================================================
> > > > Category: Unfiled
> > > >
> > > > The App-Name manual
> > > >
> > > > Include meta information from the OMF here (SK ID = MD5)
> > > >
> > > > Formats: Local HTML (split), PDF, Remote HTML (split)
> > > > =========================================================================
> > >
> > > This may be useful for people who don't have a help-browser aside from
> > > Netscape. Although Netscape won't know how to display most docs.
> >
> > True. However, we should allow the Index builder flexibilit in deciding
> > what documents are included in the Index. This may include producing a
> > special Index that only contains documents available in HTML, plain
> > text, and PDF (for example).
>
> What do you mean by "index"? The definition I gave for "index" in prop#3
> is what you find in the back of a book - an alphabetical list of concepts
> and links to where they are discussed in the given work. My impression is
> that you mean the Contents List here. Is that right?
>
> I agree that we will want searching/filtering features in sk that allow
> people to control its behavior as you suggest.
>
> Dan
>
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel
|
|
From: Ali A. <ali...@au...> - 2000-11-24 00:32:35
|
* Laszlo Kovacs (las...@su...) wrote at 20:07 on 23/11/00: > Guys, > > You have to cut back on the email size, otherwise nobody is going to > contribute to this discussion. AGREED!!!!!!!!! |
|
From: Nik C. <ni...@no...> - 2000-11-26 15:25:42
|
On Wed, Nov 22, 2000 at 09:55:45PM -0600, Dan Mueth wrote:
> > Can you give me an example of non-relocatable package, and it's
> > justification for being non-relocatable?
>
> On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> gives me 506 packages which are non-relocateable, the last of which is
> xbill:)
>
> I think Red Hat would be better qualified at explaining this decision.
Someone else bought up Gnome as an example of non-relocatable package.
Just because the original implementors didn't provide a hook to make it
non-relocatable, don't assume that other's won't have done.
> I am still uncertain if we want inheritance between contents lists though.
>
> In many cases users will not want inheritance, because they lose any
> control over what they inherit.
"many cases" means, therefore, that there will be times when users will
want inheritence.
There's a saying that's prevalent in the BSD world; "mkae tools, not policy".
Having read some of the later messages in this thread, I'm going to be
coming back to that quite a lot.
I don't care what I, personally, might need some sk. However, it's
important not to impose artificial limitations that impose my (or your)
policy for how to configure a host/network on someone else.
> We definitely want libsk.so. I'm not sure if we would want a command-line
> program for this.
It's not something we need (if someone else needs it then they can write
it). It'll probably prove to be a useful small sample application
though.
<snip>
> Aside from rebuilding the database in cases where it becomes corrupt, I'd
> like this to be a zero maintenance system that works out of the box. I
> don't think we should focus on unusual things a SA may want to
> do.
"Tools, not policy" (see, I said I'd be coming back to it). The things
that you are I think are unusual might be standard practice for someone
who wants to use sk.
> > So if you have multiple Contents Lists, you need to have multiple ToCs
> > (and Indexes), and a way of saying that >this< Contents List was used to
> > produce >This< ToC.
>
> Not if only one contents list/toc pair live in each database directory.
Tools, not policy.
I might want to put the Contents List in one directory (which is
private, and not available to the world), and publish a ToC.
We can certainly say
If you don't tell it otherwise, sk will generate the ToC in the same
place as the Contents List.
but we should make it possible for the SA to change this behaviour if
necessary.
> > Do you agree that making the distinction between a document, and one or
> > more instances of that document (different formats, different languages)
> > is worthwhile.
>
> The only way scrollkeeper knows that two docs of different
> versions/formats/languages are otherwise the same is using the extra OMF
> element (which doesn't yet exist) which assigns them an ISBN-type code
> which is the same. Since they have different IDENTIFIERS (and other
> metadata), sk knows they are different documents.
This is (possibly) something that can be handled outside of sk with
creative use of the RELATION element in the OMF.
> > > Why do we specify the format on the command line? The FORMAT is specified
> > > in the OMF already. The packager should just install and register the
> > > metadata instead of putting it on the command line.
> >
> > Because at this point, all we've done is install a document with no OMF.
> > SK has to synthesise some OMF, based on any additional information we
> > give it.
> >
> > This separation is necessary to support documents that have no separate
> > OMF. Adding "-format" to the command line is probably going to be
> > easier for an end user than having to create a separate file to return
> > to.
>
> Perhaps we agree on this after all and are just talking past each
> other. So for docs with OMF we have two steps:
> 1) update the IDENTIFIER to point to the right URL
> 2) register the OMF file with sk
>
> Really, #1 is only necessary for local docs (as opposed to remote
> ones) and for relocatable packages.
I think that's right. In my example, "sk-install-doc" is probably the
wrong name.
Ah ha, I think I've got a work around.
We've been going back and forth about what to do if a document doesn't
have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
wrong, or whatever.
So, what if we have two command, sk-generate-omf and sk-install-omf? The
usage for this would look something like;
sk-generate-omf [--omf:creator ...]
[--omf:maintainer ...]
[--omf:contributor ...]
[--omf:title ...]
[--omf:date ...]
[<other --omf:... options]
file.omf
sk-install-omf [--omf-dir ...] /path/to/omf/file
sk-generate-omf is used to generate OMF for installation into sk. It can be
told about an existing OMF file (for example, if the document has been
distributed with an OMF file). However, it also has a series of --omf:<foo>
options, which are used to override the OMF in the .omf file, and replace
it with something else (I haven't listed them all in the synopsis above).
sk-install-omf is used to install an OMF file into SK.
In the simplest case (where the document already has a separate OMF
file), in the installation Makefile, this might look something like
PREFIX= /usr/local
APP= appname
SK_OMF_DIR= `sk-config --omf-dir`
...
install-doc:
cp ${APP}.html ${PREFIX}/share/doc/${APP}.html
sk-generate-omf --omf:identifier \
${PREFIX}/share/doc/${APP}.html ${APP}.omf
sk-install-omf --omf-dir ${SK_OMF_DIR} ${APP}.omf
All this does is take the existing ${APP}.omf file, and uses
sk-generate-omf to rewrite the IDENTIFIER in the OMF file to make sure
it points to wherever the file has been installed.
This (AFAICS) completely solves the problem of relocatable packages. We
just provide a command line interface for editing OMF files.
For documents that have no OMF (i.e., the author didn't provide one),
sk-generate-omf can be used to generate one that SK can use.
> > > Right now we just number them, which guarantees a unique identifier.
> >
> > Two problems with this:
> >
> > 1. Probably won't scale efficiently as the Contents List gets larger.
> >
> > 2. Makes it impossible to merge two separate Contents Lists, from two
> > or more sites (say you're on a univesity campus, where the biology
> > department and the chemistry department both make available their
> > list of documentation in two different Scroll Keeper Contents Lists).
>
> When sk merges seperate databases, it can easily mark them so that it
> knows where the pieces came. One way to do this would be to append a few
> digit code to the beginning of each id which describes which database it
> came from.
Yeah, but that's going to needlessly complicate the mechanism for
merging databases.
The Contents List is in XML, right. XML already has a mechanism for
saying that a file includes other files.
<?xml version="1.0">
<!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
<!ENTITY contents.bio SYSTEM "/path/to/biology/contents/list">
<!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">
]>
<contents>
&contents.bio;
&contents.chem;
</contents>
Why reinvent the wheel? All we need to do is ensure (as reasonably as
possible) that the identifiers in the different contents lists are
unique.
> > Your document browser is going to need a mechanism to uniquely
> > identify each document. If there's a strong probability that the
> > document IDs are not unique then the browsers are each going to
> > have to reinvent their own mechanism for ensuring that the
> > documents are unique.
> >
> > The documents URL might be appropriate, but it's a bit unwieldy.
>
> The document URL is a very good unique identifier, and it is already being
> passed to the help browser.
Yes, it's probably the best thing to use.
> > You really need a mechanism that can (as near as possible) guarantee
> > globally unique document IDs.
>
> I'm pretty sure that only one document lives at a given URL ;)
Not necessarily. Web servers have been known to serve up different
document depends on the settings of the browser requesting the document
(for example, it's list of acceptable languages).
> > True. However, we should allow the Index builder flexibilit in deciding
> > what documents are included in the Index. This may include producing a
> > special Index that only contains documents available in HTML, plain
> > text, and PDF (for example).
>
> What do you mean by "index"? The definition I gave for "index" in prop#3
> is what you find in the back of a book - an alphabetical list of concepts
> and links to where they are discussed in the given work. My impression is
> that you mean the Contents List here. Is that right?
Yes, my mistake.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Ali A. <ali...@au...> - 2000-11-27 14:32:32
|
* Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
> On Wed, Nov 22, 2000 at 09:55:45PM -0600, Dan Mueth wrote:
> > > Can you give me an example of non-relocatable package, and it's
> > > justification for being non-relocatable?
> >
> > On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> > gives me 506 packages which are non-relocateable, the last of which is
> > xbill:)
> >
> > I think Red Hat would be better qualified at explaining this decision.
>
> Someone else bought up Gnome as an example of non-relocatable package.
>
> Just because the original implementors didn't provide a hook to make it
> non-relocatable, don't assume that other's won't have done.
This is not scrollkeeper's problem though. We should support both instead of
imposing relocatable packages on people.
Actually, I think if we supposed relocatable packages, we will (by definition)
have supported non-relocatable packages, so this is a moot point.
> > I am still uncertain if we want inheritance between contents lists though.
> >
> > In many cases users will not want inheritance, because they lose any
> > control over what they inherit.
>
> "many cases" means, therefore, that there will be times when users will
> want inheritence.
>
> There's a saying that's prevalent in the BSD world; "mkae tools, not policy".
> Having read some of the later messages in this thread, I'm going to be
> coming back to that quite a lot.
>
> I don't care what I, personally, might need some sk. However, it's
> important not to impose artificial limitations that impose my (or your)
> policy for how to configure a host/network on someone else.
I don't see how scrollkeeper is imposing policy. We want it to support both
relocatable and non-relocatable packages.
It seems that you want to to "impose" the policy of relocatable packages only
(although the concept of relocatable packages is policy imposing).
> > We definitely want libsk.so. I'm not sure if we would want a command-line
> > program for this.
>
> It's not something we need (if someone else needs it then they can write
> it). It'll probably prove to be a useful small sample application
> though.
Agreed. This shouldn't be one of the "targets" for 1.0, but if somebody does
it we could include it. Also, if post-1.0 comes and it is deemed a
command-line tool is needed, then we can worry about it
> <snip>
>
> > Aside from rebuilding the database in cases where it becomes corrupt, I'd
> > like this to be a zero maintenance system that works out of the box. I
> > don't think we should focus on unusual things a SA may want to
> > do.
>
> "Tools, not policy" (see, I said I'd be coming back to it). The things
> that you are I think are unusual might be standard practice for someone
> who wants to use sk.
I think the goal is to make scrollkeeper "automated" enough so that Desktop
users can use it (remember - Help Documents are (usually) for new users).
Now if a sysadmin wanted to play around and mess with ScrollKeeper, I think
this also should be possible. Now, what kind of things would a sysadmin want
to do that should be supported?
> > > So if you have multiple Contents Lists, you need to have multiple ToCs
> > > (and Indexes), and a way of saying that >this< Contents List was used to
> > > produce >This< ToC.
> >
> > Not if only one contents list/toc pair live in each database directory.
>
> Tools, not policy.
>
> I might want to put the Contents List in one directory (which is
> private, and not available to the world), and publish a ToC.
>
> We can certainly say
>
> If you don't tell it otherwise, sk will generate the ToC in the same
> place as the Contents List.
>
> but we should make it possible for the SA to change this behaviour if
> necessary.
I really don't think this is a major issue (to specify which directory a file
goes into).
This is one of the minor details that can be taken care of.
The issue here is: support multiple ToCs and Contents Lists.
> > > Do you agree that making the distinction between a document, and one or
> > > more instances of that document (different formats, different languages)
> > > is worthwhile.
> >
> > The only way scrollkeeper knows that two docs of different
> > versions/formats/languages are otherwise the same is using the extra OMF
> > element (which doesn't yet exist) which assigns them an ISBN-type code
> > which is the same. Since they have different IDENTIFIERS (and other
> > metadata), sk knows they are different documents.
>
> This is (possibly) something that can be handled outside of sk with
> creative use of the RELATION element in the OMF.
I've actually never seen an OMF file *goes and hangs his head in shame*
> > > > Why do we specify the format on the command line? The FORMAT is specified
> > > > in the OMF already. The packager should just install and register the
> > > > metadata instead of putting it on the command line.
> > >
> > > Because at this point, all we've done is install a document with no OMF.
> > > SK has to synthesise some OMF, based on any additional information we
> > > give it.
> > >
> > > This separation is necessary to support documents that have no separate
> > > OMF. Adding "-format" to the command line is probably going to be
> > > easier for an end user than having to create a separate file to return
> > > to.
> >
> > Perhaps we agree on this after all and are just talking past each
> > other. So for docs with OMF we have two steps:
> > 1) update the IDENTIFIER to point to the right URL
> > 2) register the OMF file with sk
> >
> > Really, #1 is only necessary for local docs (as opposed to remote
> > ones) and for relocatable packages.
>
> I think that's right. In my example, "sk-install-doc" is probably the
> wrong name.
>
> Ah ha, I think I've got a work around.
>
> We've been going back and forth about what to do if a document doesn't
> have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
> wrong, or whatever.
>
> So, what if we have two command, sk-generate-omf and sk-install-omf? The
> usage for this would look something like;
>
> sk-generate-omf [--omf:creator ...]
> [--omf:maintainer ...]
> [--omf:contributor ...]
> [--omf:title ...]
> [--omf:date ...]
> [<other --omf:... options]
> file.omf
>
> sk-install-omf [--omf-dir ...] /path/to/omf/file
>
> sk-generate-omf is used to generate OMF for installation into sk. It can be
> told about an existing OMF file (for example, if the document has been
> distributed with an OMF file). However, it also has a series of --omf:<foo>
> options, which are used to override the OMF in the .omf file, and replace
> it with something else (I haven't listed them all in the synopsis above).
Umm - okay, lets say I have a well-formed OMF file and then I do
'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?
Hmm, you seem to be implying that people will not write OMF files and that
they should be "extracted" from the docs. From my understanding, they are a
part of the doc (there was discussion on wether to include necessary tags in
the docs themselves, or to have separate OMF files).
I think perhaps we should just allow the '--omf:' overrides in the
sk-install-omf binary
> sk-install-omf is used to install an OMF file into SK.
>
> In the simplest case (where the document already has a separate OMF
> file), in the installation Makefile, this might look something like
>
> PREFIX= /usr/local
> APP= appname
> SK_OMF_DIR= `sk-config --omf-dir`
>
> ...
>
> install-doc:
> cp ${APP}.html ${PREFIX}/share/doc/${APP}.html
> sk-generate-omf --omf:identifier \
> ${PREFIX}/share/doc/${APP}.html ${APP}.omf
> sk-install-omf --omf-dir ${SK_OMF_DIR} ${APP}.omf
>
> All this does is take the existing ${APP}.omf file, and uses
> sk-generate-omf to rewrite the IDENTIFIER in the OMF file to make sure
> it points to wherever the file has been installed.
>
> This (AFAICS) completely solves the problem of relocatable packages. We
> just provide a command line interface for editing OMF files.
>
> For documents that have no OMF (i.e., the author didn't provide one),
> sk-generate-omf can be used to generate one that SK can use.
I think at first, ScrollKeeper will only support DocBook SGML/XML (which I
think is a decent goal for '1.0' (then from the experience with that we can
possible tackle other (more difficult) file formats)).
If a document does not have an OMF file, we should not attempt to "guess"
anything in any way. This document should not be cataloged/indexed.
ScrollKeeper is here to handle /OMF/ files, if they are not present then
obviously you shouldn't use Scrollkeeper. Even if we try to "guess" the
information, we can't figure out things like Author name, language, category,
version #, document title, etc. (esp. if ScrollKeeper is to handle things like
HTML or plain text).
> > > > Right now we just number them, which guarantees a unique identifier.
> > >
> > > Two problems with this:
> > >
> > > 1. Probably won't scale efficiently as the Contents List gets larger.
> > >
> > > 2. Makes it impossible to merge two separate Contents Lists, from two
> > > or more sites (say you're on a univesity campus, where the biology
> > > department and the chemistry department both make available their
> > > list of documentation in two different Scroll Keeper Contents Lists).
> >
> > When sk merges seperate databases, it can easily mark them so that it
> > knows where the pieces came. One way to do this would be to append a few
> > digit code to the beginning of each id which describes which database it
> > came from.
>
> Yeah, but that's going to needlessly complicate the mechanism for
> merging databases.
>
> The Contents List is in XML, right. XML already has a mechanism for
> saying that a file includes other files.
>
> <?xml version="1.0">
> <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
>
> <!ENTITY contents.bio SYSTEM "/path/to/biology/contents/list">
> <!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">
>
> ]>
>
> <contents>
> &contents.bio;
>
> &contents.chem;
> </contents>
>
> Why reinvent the wheel? All we need to do is ensure (as reasonably as
> possible) that the identifiers in the different contents lists are
> unique.
Actually - this is a VERY good idea. In fact, DV just suggested this idea on
IRC which I didn't quite understand (then I read this example and I get it)...
Basically, you can have a /etc/scrollkeeper.conf file that lists all the
directories which Scrollkeeper will scan and include as entities.
I have a question though. How will this work for user's directories? Does
Scrollkeeper need to support this "indepenendently (and track it with a
~/.scrollkeeperrc)
> > > Your document browser is going to need a mechanism to uniquely
> > > identify each document. If there's a strong probability that the
> > > document IDs are not unique then the browsers are each going to
> > > have to reinvent their own mechanism for ensuring that the
> > > documents are unique.
> > >
> > > The documents URL might be appropriate, but it's a bit unwieldy.
> >
> > The document URL is a very good unique identifier, and it is already being
> > passed to the help browser.
>
> Yes, it's probably the best thing to use.
URLs are very unreliable and subject to change actually. There are not fixed
forever. You can just move them to another directory and "poof". Servers can
shutdown, etc.
[snipped the rest in an effor to conserve size]
|
|
From: Nik C. <ni...@no...> - 2000-12-04 20:10:17
|
On Mon, Nov 27, 2000 at 03:42:38AM -0200, Ali Abdin wrote:
> * Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
> > Just because the original implementors didn't provide a hook to make it
> > non-relocatable, don't assume that other's won't have done.
>
> This is not scrollkeeper's problem though. We should support both instead of
> imposing relocatable packages on people.
>
> Actually, I think if we supposed relocatable packages, we will (by definition)
> have supported non-relocatable packages, so this is a moot point.
Ah ha. Someone gets it :-) That's right exactly.
> > > Aside from rebuilding the database in cases where it becomes corrupt, I'd
> > > like this to be a zero maintenance system that works out of the box. I
> > > don't think we should focus on unusual things a SA may want to
> > > do.
> >
> > "Tools, not policy" (see, I said I'd be coming back to it). The things
> > that you are I think are unusual might be standard practice for someone
> > who wants to use sk.
>
> I think the goal is to make scrollkeeper "automated" enough so that Desktop
> users can use it (remember - Help Documents are (usually) for new users).
>
> Now if a sysadmin wanted to play around and mess with ScrollKeeper, I think
> this also should be possible. Now, what kind of things would a sysadmin want
> to do that should be supported?
It's not a "What would they want to do" thing. It's a philosophy of not
building in restrictions (like "There will only ever be one contents
list") in to the system. We might not be able to immediately see why
multiple contents lists would be useful. But the extra effort in
supporting them is useful.
<snip>
> > but we should make it possible for the SA to change this behaviour if
> > necessary.
>
> I really don't think this is a major issue (to specify which directory a file
> goes into).
It's not a major issue, no. What I'm trying to do is highlight where
people are (or could be) inadvertently hardcoding in design decisions
that haven't been thought through.
If we have a feature list that we agree with, and one of those features
is "sk will support multiple contents lists" then it's unlikely that
we'll forget that as we implement sk.
I haven't seen this sort of design document yet, and the "DD:" points in
my original message were supposed to seed that, so that we can agree on
some common functionality, irrespective of the specific implementation
details.
[ Quote kept for context ]
> > > > > Why do we specify the format on the command line? The FORMAT is specified
> > > > > in the OMF already. The packager should just install and register the
> > > > > metadata instead of putting it on the command line.
> > > >
> > > > Because at this point, all we've done is install a document with no OMF.
> > > > SK has to synthesise some OMF, based on any additional information we
> > > > give it.
> > > >
> > > > This separation is necessary to support documents that have no separate
> > > > OMF. Adding "-format" to the command line is probably going to be
> > > > easier for an end user than having to create a separate file to return
> > > > to.
> > >
> > > Perhaps we agree on this after all and are just talking past each
> > > other. So for docs with OMF we have two steps:
> > > 1) update the IDENTIFIER to point to the right URL
> > > 2) register the OMF file with sk
> > >
> > > Really, #1 is only necessary for local docs (as opposed to remote
> > > ones) and for relocatable packages.
> >
> > I think that's right. In my example, "sk-install-doc" is probably the
> > wrong name.
> >
> > Ah ha, I think I've got a work around.
> >
> > We've been going back and forth about what to do if a document doesn't
> > have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
> > wrong, or whatever.
> >
> > So, what if we have two command, sk-generate-omf and sk-install-omf? The
> > usage for this would look something like;
> >
> > sk-generate-omf [--omf:creator ...]
> > [--omf:maintainer ...]
> > [--omf:contributor ...]
> > [--omf:title ...]
> > [--omf:date ...]
> > [<other --omf:... options]
> > file.omf
> >
> > sk-install-omf [--omf-dir ...] /path/to/omf/file
> >
> > sk-generate-omf is used to generate OMF for installation into sk. It can be
> > told about an existing OMF file (for example, if the document has been
> > distributed with an OMF file). However, it also has a series of --omf:<foo>
> > options, which are used to override the OMF in the .omf file, and replace
> > it with something else (I haven't listed them all in the synopsis above).
>
> Umm - okay, lets say I have a well-formed OMF file and then I do
> 'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?
Nothing. But if you do
sk-generate-omf --omf:creator 'Foo Bar <fo...@ex...>' file.omf
then sk-generate-omf has just rewritten the content of the CREATOR
element in file.omf.
sk-generate-omf < file.omf > file2.omf
is probably the more natural way of running it though.
> Hmm, you seem to be implying that people will not write OMF files and that
> they should be "extracted" from the docs. From my understanding, they are a
> part of the doc (there was discussion on wether to include necessary tags in
> the docs themselves, or to have separate OMF files).
Not quite. I think we need to cater for a few sets of eventualities.
1. No OMF provided, but document is in an OMF'able format, or where
some of the OMF can be extracted.
2. No OMF provided, and document is not in an OMF'able format (such
as plain text)
3. OMF provided, but it's incorrect in some way.
4. OMF provided, and it's correct.
In the case of (1) I think we need some tools that can extract OMF from
common formats (DocBook, man, LinuxDoc, ...). Whilst these tools are
not strictly part of sk itself (anyone want to start an omf-tools
project?) I think they'll be important to aid the take up of sk.
In 1, we'll also need the sk-generate-omf command to update the OMF
after it's been extracted from the source document.
In the case of (2) we'll need an sk-generate-omf command to make it easy
for the end user to generate valid OMF files.
[ To anticipate your question -- "Why not get the end user to create the
OMF files using a text editor?", I think the reason is three-fold.
First, the user doesn't need to learn a new text format.
Second, this makes it impossible for the user to create an invalid OMF
file.
Third, it insulates the end user from possible changes in the OMF file
format (i.e., the XML DTD). ]
In the case of (3), we'll need sk-generate-omf to fixup the incorrect
OMF data.
In the case of (4), we don't need to do anything.
> I think perhaps we should just allow the '--omf:' overrides in the
> sk-install-omf binary
I disagree. Don't have one tool that does multiple things, have
multiple tools that do one thing each. It's useful to be able to
generate OMF without forcing the end user to install the documentation
as part of the process.
<snip>
> > For documents that have no OMF (i.e., the author didn't provide one),
> > sk-generate-omf can be used to generate one that SK can use.
>
> I think at first, ScrollKeeper will only support DocBook SGML/XML (which I
> think is a decent goal for '1.0' (then from the experience with that we can
> possible tackle other (more difficult) file formats)).
>
> If a document does not have an OMF file, we should not attempt to "guess"
> anything in any way.
Agreed. But tools that make it easier for the end user to have a stab
at generating OMF would be useful. For example, it would greatly ease
the end user's job if they can do something like
cd /usr/share/man/man1
foreach i (*.1)
man2omf $i > /usr/local/share/omf/man/$i.omf
end
This doesn't mean that the generated OMF files are 100% correct (note:
not 'valid', as man2omf should always generate valid XML) -- for
example, the author information might say "Automatically generated by
man2omf", or some such.
As I say earlier though, this is not strictly sk's responsibility.
However, I suggest that development of small helper apps like these will
increase the speed with which people can usefully use sk on their
system.
> > The Contents List is in XML, right. XML already has a mechanism for
> > saying that a file includes other files.
> >
> > <?xml version="1.0">
> > <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
> >
> > <!ENTITY contents.bio SYSTEM "/path/to/biology/contents/list">
> > <!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">
> >
> > ]>
> >
> > <contents>
> > &contents.bio;
> >
> > &contents.chem;
> > </contents>
> >
> > Why reinvent the wheel? All we need to do is ensure (as reasonably as
> > possible) that the identifiers in the different contents lists are
> > unique.
>
> Actually - this is a VERY good idea.
Woo hoo
[ Sorry, too much Simpsons ]
> In fact, DV just suggested this idea on
> IRC which I didn't quite understand (then I read this example and I get it)...
>
> Basically, you can have a /etc/scrollkeeper.conf file that lists all the
> directories which Scrollkeeper will scan and include as entities.
Not quite. XML can do all this for you.
Suppose, for example, that the sysadmin wants to make it easy for end
users to create their own contents lists that include the system's
installed documentation.
They might create a default Contents List that looks like this:
<?xml version="1.0">
<!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
<!ENTITY contents.system SYSTEM "/path/to/system/contents/list">
<!ENTITY contents.user SYSTEM "&HOME;/.sk/contents/list">
]>
<contents>
&contents.system;
&contents.user;
</contents>
There's only one peice of 'magic' in here, "&HOME;".
Basically, what I'm trying to show here is that this master contents
list would pull in the contents of /path/to/system/contents/list, and
would then pull in /home/$USER/.sk/contents/list.
So if the user goes and creates their own contents list in
~/.sk/contents/list and runs sk then they're changes are automatically
picked up.
Unfortunately, XML (and SGML) has no concept of using environment
variables inside paths like this.
What you could do is hardcode sk to recognise $HOME inside a Contents
List and automatically expand it to the appropriate environment
variable. However, that's a bit of a kludge, and also takes us outside
of XML functionality.
However, it's not too much of a stretch to say (and document) that when
SK parses a Contents List it pre-initialises a few entities for use in
the Contents List. One of those might be &HOME;, which corresponds with
$HOME, another might be &LC_LANG;, which corresponds with $LC_LANG, and
so on.
Actually, that's a pretty cool idea. You could imagine a master
Contents List that looks like
<?xml version="1.0">
<!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
<!ENTITY contents SYSTEM "/var/db/sk/&LC_LANG;/contents">
]>
<contents>
&contents;
</contents>
which would pick up the contents list from /var/db/sk/$LC_LANG/contents
depending on the setting of the $LC_LANG environment variable of the
person running sk.
> I have a question though. How will this work for user's directories? Does
> Scrollkeeper need to support this "indepenendently (and track it with a
> ~/.scrollkeeperrc)
See above. This approach actually lets the SA create a global config
file that automatically pulls in $HOME/.sk/, without the user needing to
do anything special. The name of the file it pulls in is completely at
the discretion of the SA.
Of course, sk would ship with defaults that show how all this works.
N
--
Internet connection, $19.95 a month. Computer, $799.95. Modem, $149.95.
Telephone line, $24.95 a month. Software, free. USENET transmission,
hundreds if not thousands of dollars. Thinking before posting, priceless.
Somethings in life you can't buy. For everything else, there's MasterCard.
-- Graham Reed, in the Scary Devil Monastery
|
|
From: Gregory L. <gle...@cu...> - 2000-12-04 22:15:19
|
> On Mon, Nov 27, 2000 at 03:42:38AM -0200, Ali Abdin wrote:
> > * Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
>
> I haven't seen this sort of design document yet, and the "DD:" points in
> my original message were supposed to seed that, so that we can agree on
> some common functionality, irrespective of the specific implementation
> details.
Didn't somebody have these? I could have sworn I'd seen them, but I
can't seem to figure out where.
> > > sk-generate-omf is used to generate OMF for installation into sk. It can be
> > > told about an existing OMF file (for example, if the document has been
> > > distributed with an OMF file). However, it also has a series of --omf:<foo>
> > > options, which are used to override the OMF in the .omf file, and replace
> > > it with something else (I haven't listed them all in the synopsis above).
> >
> > Umm - okay, lets say I have a well-formed OMF file and then I do
> > 'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?
>
> Nothing. But if you do
>
> sk-generate-omf --omf:creator 'Foo Bar <fo...@ex...>' file.omf
>
> then sk-generate-omf has just rewritten the content of the CREATOR
> element in file.omf.
>
> sk-generate-omf < file.omf > file2.omf
>
> is probably the more natural way of running it though.
>
> > Hmm, you seem to be implying that people will not write OMF files and that
> > they should be "extracted" from the docs. From my understanding, they are a
> > part of the doc (there was discussion on wether to include necessary tags in
> > the docs themselves, or to have separate OMF files).
>
> Not quite. I think we need to cater for a few sets of eventualities.
>
> 1. No OMF provided, but document is in an OMF'able format, or where
> some of the OMF can be extracted.
>
> 2. No OMF provided, and document is not in an OMF'able format (such
> as plain text)
>
> 3. OMF provided, but it's incorrect in some way.
>
> 4. OMF provided, and it's correct.
>
> In the case of (1) I think we need some tools that can extract OMF from
> common formats (DocBook, man, LinuxDoc, ...). Whilst these tools are
> not strictly part of sk itself (anyone want to start an omf-tools
> project?) I think they'll be important to aid the take up of sk.
I've borrowed a perl hack from Greg Ferguson, and I've made (and had
made) a few changes to it. However, there are a few things that don't
quite work just yet, I'll let people know when I've got tools working.
> In 1, we'll also need the sk-generate-omf command to update the OMF
> after it's been extracted from the source document.
Why does it need to be updated immediately after it's been
extracted? Or did I miss something?
> In the case of (2) we'll need an sk-generate-omf command to make it easy
> for the end user to generate valid OMF files.
>
> [ To anticipate your question -- "Why not get the end user to create the
> OMF files using a text editor?", I think the reason is three-fold.
>
> First, the user doesn't need to learn a new text format.
>
> Second, this makes it impossible for the user to create an invalid OMF
> file.
>
> Third, it insulates the end user from possible changes in the OMF file
> format (i.e., the XML DTD). ]
I don't think this is something that should be part of sk, per say, but
some "extra" module.
> In the case of (3), we'll need sk-generate-omf to fixup the incorrect
> OMF data.
That sounds like fun, I'm very familiar with "automagically" fixing XML
documents, based on the errors that a parser spits out. I'd guess that
the w3c's Tidy application would contain some of the concepts needed for
this.
> > I think perhaps we should just allow the '--omf:' overrides in the
> > sk-install-omf binary
>
> I disagree. Don't have one tool that does multiple things, have
> multiple tools that do one thing each. It's useful to be able to
> generate OMF without forcing the end user to install the documentation
> as part of the process.
Yes, one tool that does one thing is "The Unix Way", and I think it's
probably a good way to go. Would someone want to have this available at
the time that they install the package? If so, it needs to be part of
sk.
Greg
|
|
From: Ali A. <ali...@au...> - 2000-11-24 00:32:40
|
* Nik Clayton (ni...@no...) wrote at 19:17 on 23/11/00:
> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> >
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
>
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?
GNOME is non-relocatable...Nautilus is non-relocatable (I maybe incorrect, somebody
told me they were non-relocatable)
The reason is the "datadirs" are define'd at compiletime.
Lets take an example....I have an application called gLife that uses a glade
file for its UI. The glade file is installed in $(prefix)/share/glife (prefix
is defined at ./configure time)....so to load the glade file, I need the
/path/ to the glade file. This is defined during compilation as GLIFE_DATADIR.
Now, if you compile the application into an RPM for '/usr' then you try
installing it in '/opt', the application will still look for the glade file in
/usr/share/glife.
Now, I know there are probably hacks that can be done to get around this. I
know that this may not be "theoretically" correct, etc. etc. etc. But the fact
is there /ARE/ non-relocatable packages out there, and scrollkeeper should
support both types of packages.
> > The current way things are designed is that it will have one contents list
> > for the system.
>
> I think that's a bad idea.
>
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does. I'm describing
> what I think SK should do, and what the interfaces should be. It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
>
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed. By specifying which OMF directories you want to include,
> > these databases could be made to differ. We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned. This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
>
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.
I agree, they should be separate.
> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization. The problem is
> > that somebody has to specify what each database contains. So it works for
> > users, because they can hand-tweak the contents. It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager. So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.
>
> I think it does, as long as a mechanism is provided so that sk can work
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
>
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF. OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
>
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.
>
> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document. The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.
>
> Also, consider a regular user who occasionally installs applications and
> documentation in to $HOME.
>
> It's much easier for them to add a flag to a command line (or, better
> still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
> than it is for them to edit OMF files before installing them.
>
> > What is the advantage of breaking the main database into pieces?
>
> Flexibility for the future.
There should be support in mind for "multiple" databases in my opinion. This
way, a person can set an environment flag, or a command-line option and be
able to load an entirely new database (along with his old one).
Perhaps you should have a '/etc/scrollkeeper.conf' file that has a list of
various databases as well as various configuration options.
In addition to this, I think scrollkeeper should handle any databases found in
'~/.scrollkeeper/'
So, multiple databases does make sense in my opinion (think of it this way,
just because YOU don't need it, or can't find a reason to use it, it doesn't
mean that it is not useful for others)..
From a desktop perspective it might not be necessary to have multiple
databases, but from a "sysadmin" perspective it does.
> > > 2. Maintaining a TOC from the Contents List
> > >
> > > 3. Maintaining an Index from the Contents List
> > >
> > > 4. Maintaining a Categories list
> >
> > Well, this is somewhat debatable and worthy of discussion. Somebody has
> > to provide and maintain the categories list. On one side, we can just
> > define it as part of OMF/sk and install a static list with sk. Then no
> > maintenance is necessary. Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> > sys admin could drop in a replacement if they want. For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
>
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist. I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.
FAQ/Manual/HOWTO are not category types in your opinion...What I mean by
categories is more like "subject" - i.e. "Audio/MP3 Players' or
'Office/Spreadsheet' or 'X11/Internet/FTP Clients' etc.
The fact that it is a FAQ/HOWTO/Manual should be reflected in the name of the
document (I think this is the "purpose" of a document).
There should also be another field that specifies the type of document (HTML,
Info, Man, DocBook, plain text, etc.)
> However, that's not the same thing as saying
>
> If a categories file exists (in some, as yet undecided format), SK
> should have the option of warning the SA that they're about to install
> a document with a category that does not currently exist, and asking
> whether they want to continue. Naturally, the SA should have the
> option of overriding this check with a command line flag.
I am saying just a HTML file, or plain text file just LISTING the categories -
nothing fancy...Scrollkeeper would have no knowledge of this file and will not
interact with it in anyway.
The scrips should NEVER EVER become "interactive". Just use whatever is in the
category (in the OMF file).
> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > application will have the responsibility for handling user queries of the
> > > SK TOC and Index files ]
> >
> > sk should export searching functionality.
>
> How? libsk.so? By having a program that accepts search parameters and
> returns XML with the results? Some other mechanism?
I prefer a library that returns the xmlDocPtr of the file...you can then parse
it as you wish. Note: with a library you could _EASILY_ create a command-line
application that would just "dump" the XML tree to stdout (if you need it)
> > > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > > maintain several different TOCs and Indices from several different Contents
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> >
> > For a given database, I agree. Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one. Is
> > this what you have in mind? If you think we want more than one contents
> > list in the system's database, please explain why.
>
> Flexibility and efficiency. Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on. That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.
>
> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man. So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.
I think this "optimization" is unncessary at this time. We have no data _AT
ALL_ on the efficiency in scrollkeeper. We don't know if it will be slow or
fast, so splitting up databases for this reason is unacceptable in my opinion.
On the other hand, I do think we should support multiple databases (through a
command-line option, and possible an environment variable (and API functions
when the library is done))
> > > What sort of documentation can the Contents List going to point to?
> > >
> > > 1. Locally installed standard documentation that comes with the system,
> > > such as man pages, or Info documents.
> >
> > Perhaps. I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below. For man and info, we would have to update this information
> > off of cron unfortunately. So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
>
> I don't quite agree. As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
>
> First, the user tells SK about the documentation.
>
> Second, the user assigns some OMF to the documentation.
>
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.
>
> Consider the man page, for example. If I want to tell SK about all my
> section 1 man pages, I'd like to be able to do something like
>
> cd /usr/share/man/man1
> sk-install-doc *.1
>
> which would tell SK about all my section 1 manual pages. Actually, what
> I'd probably do is something like
>
> sk-install-doc -category "Man Pages:Section 1" *.1
>
> to assign a category to them. At this point, there's no OMF for them,
> but SK can synthesise the bare bones from the filename and the category.
>
> Suppose, then, that I want to add some OMF for ls.1. I might do
> something like this;
>
> sk-extract-omf ls.1 > ls.omf
> vi ls.omf
> [edit edit edit]
> sk-install-omf ls.1 ls.omf
>
> which would pull out the OMF that SK synthesised for me in the earlier
> step (presumably as XML). I can then edit it, and then install the
> edited OMF back in to SK's control.
>
> I think it's highly likely that *roff -man (and in particular, *roff
> -mdoc) is going to be easy to extract OMF from. So what we might end up
> with is a command to extract OMF from -man formatted documents. It
> might work like this;
>
> cd /usr/share/man/man1
> foreach man in (*.1)
> extract-omf -source man $man > `basename $man .1`.omf
> end
>
> which would iterate over *.1, extracting the OMF and writing it to
> *.omf. With that done, the end user might then be able to do
>
> foreach man in (*.1)
> sk-install-doc -doc $man -omf `basename $man .1`.omf
> end
>
> which could install the document, and the pregenerated OMF, into SK with
> one command.
>
> I hope the design ethos that's coming across from all of this is to have
> lots of small tools, each of which does one thing.
I disagree here...Without the OMF data ScrollKeeper shouldn't do jack shit to
the document. It shouldn't even know the document exists.
If somebody wishes to write an application that would take a document, create
an OMF file from the document, and register it with ScrollKeeper - well, that
would be nice. But its not /ScrollKeepers/ responsibility.
Maybe something like this can be done in the future, but I don't see this as
an 'essential feature'...
Also it would be "nice" to have a command-line tool that would allow you
modify the OMF metadata based on the command-line options.
> > > 2. Locally installed system documentation, installed by the SA team.
> >
> > SA =? sys admin
>
> Yes.
>
> > > 3. Locally installed documentation, installed by the user.
> >
> > Yes. Although like much of this discussion, this feature will come in a
> > post-0.1 release.
>
> As I say, I'm not thinking in terms of 'release' yet. I'd like to get
> some specific functionality nailed down and agreed on.
>
> [ Incidentally, and talking about the implementation -- one of the
> things that occured to me last night; sk could have a command
> structure like that of cvs, where instead of having a bunch of
> sk-foo-bar commands, you have one 'sk' command, that takes options,
> and then subcommands, which also take options.
>
> sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
> sk install-omf /path/to/omf
>
> and so on. ]
Difficult I think since the SK stuff are two distinct separate tools :) but it
would be an interesting thing to do...
> > > I think it is fair to say that, when run, sk will need to construct it's
> > > global Contents List by parsing multiple, smaller, contents lists. In much
> > > the same way that SGML catalog files can include one another at the moment.
> >
> > Why? It sounds very nice and wonderful. However, this adds a layer of
> > complexity without any obvious benefits.
>
> As above.
>
> Note that I don't mean that SK goes and writes a new, master contents
> list file. Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).
You could just re-construct one large XML tree in memory from multiple smaller
XML files. You would then conduct all operations on this in-memory XML tree.
It won't make much diff. performance wise if you do this on one LARGE xml
file, or multiple smaller ones i believe (provided you are doing the above
approach of building a large XML tree from smaller ones).
> > > DD: sk must have a run time option to allow the end user to select which
> > > TOC to update.
> >
> > The databases are stored together. So, this should be specified at the
> > same time as the contents list part of the database. I don't think I want
> > my contents list coming from one database and my toc from another.
> >
> > (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> > TOC.)
>
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC. That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.
>
> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.
>
> Or are you saying that the ToC should be stored within the Contents List?
>
> > > DD: sk must have a run time option to allow the end user to select which
> > > Index to update.
> >
> > (as above)
>
> Ditto :-)
>
> > > DD: One of the sk commands must be to generate a usable Contents List
> > > (which really means {X}HTML version, to start with) given a Contents
> > > List which is in sk internal format.
> >
> > Everything is XML right now, so this is not an issue. If we decide
> > another internal format is better we should still export XML to the
> > browser. As for HTML, this is possible and may be a nice interface for
> > people who want to create a very minimalistic web browser.
>
> All the world is not yet XML, and not everyone runs KDE or Gnome. Being
> able to convert the ToCs and Indices to other formats is vital.
> Fortunately, since the ToC and Indices are XML, this should be easy,
> using something like XSLT, so we can probably just consider this
> functionality that we get for free.
Well, even if you don't use KDE or GNOME, you would still need to parse
ScrollKeeper's file-format...
Lets say we /don't/ use XML, you would still have to handle whatever internal
data structure is used by ScrollKeeper. We just picked XML because it is
(relatively) easier to parse.
> > > DD: It is possible for a document to change, but for the OMF information
> > > to be neglected. sk should maintain a modification time stamp for
> > > each document and its corresponding OMF, to allow the end user to
> > > determine when that might have happened, and to correct for it.
> >
> > For the current version of sk, which does not yet allow for remote
> > documents, only the OMF file's timestamp is watched. This is because if
> > somebody edits the doc and not the OMF, then it is their fault for not
> > updating the OMF, not sk's. In reality whenever you install a new version
> > of a package, even if the OMF is not updated (as it should), it still
> > updates the old version so it gets a new timestamp and the new doc gets
> > registered despite any negligence on the part of the maintainer to update
> > the OMF (or the doc for that matter;).
> >
> > This mainly becomes an issue for:
> > 1) remote docs
> > 2) docs which users write and modify themselves.
> >
> > So we may indeed need to start tracking the doc timestamps when we support
> > these features. We have to be careful about constantly checking the
> > timestamps on a bunch of docs which are spread around the internet
> > though. There will be many cases where this task cannot complete, or else
> > takes an unacceptable amount of time.
>
> This should be configurable behaviour, something like;
>
> sk --check-remote-timestamps --remote-timeout=30
>
> Actually, this implies another DD
>
> DD: sk must track the validity of the OMF in relation to it's document.
> OMF might be:
>
> valid This OMF was installed with the document, and neither
> the document or the OMF's timestamps have changed.
>
> invalid The document and OMF have different timestamps
>
> unknown The document and OMF have the same timestamps, but
> the most recent attempt to confirm this (particularly
> for remote documents) failed.
>
> sk should be able to report on documents who have OMF which is not
> valid, so the SA can investigate further.
>
> > > DD: One document, multiple output formats. The user might have installed
> > > the same document three times, once as HTML, once as PDF, and once as
> > > plain text. The three installations are the same document, and have
> > > the same OMF associated with them.
> >
> > No. These definitely do not have the same OMF, unless they are identical
> > copies. The OMF file specifies the document format in FORMAT.DTD and
> > FORMAT.MIME. The docs also have different IDENTIFIER's.
> >
> > (We may decide to expand this a little bit. FORMAT.STYLESHEET may be
> > nice, as it would allow the doc authors to dictate the appearance of their
> > doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> > icon, etc.)
> >
> > > sk should treat this as one document with three formats, rather than
> > > three separate documents.
> >
> > If you look in the ldp-meta archives from several months ago, I proposed
> > we add a document id to the OMF. This would allow us to identify if two
> > documents are "The-Same". I call two documents "The-Same" if they are the
> > same document only different versions, languages, formats, etc. If two
> > documents have the same title, they are not necessarily
> > "The-Same". Likewise, if I change the title of my document at some point
> > in time, I can have two docs which are "The-Same" but not having the same
> > title.
> >
> > Obviously this sort of information will be very nice to have. You can do
> > things like "Get me this document in language xxx." or "Show me the latest
> > version of this document on the Net." or "Show me the previous version of
> > this document from the Net."
> >
> > > [ This is probably going to be the most contentious piece of this
> > > writeup, and it might change. In particular, it also suggests that
> > > translations of a document should be treated as the same document,
> > > just in a different language, and I'm not sure that's a good idea.
> > > Then again, it might be. ]
> >
> > I guess we were thinking along similar lines :)
>
> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.
Translations are considered to be "separate" documents for me.
I don't know about different formats...that is a difficult one - but why would
we need to distinguish between instances of a document based on format?
I like the idea of an "id" of a document, because then you can have diff.
versions of a doc. It is difficult to determine wether translations should
have their own id...I recommend you just flip a coin.
The good thing about an id (unique id's can be obtained using 'uuid-gen') is
that you can query any server in the globe for that particular id and it will
retain that document (provided you also provide the version number, and
possible the format)
> > > There are a number of tasks that the sk end user is going to want to tell
> > > sk to do. They are;
> > >
> > > Install new documentation that comes bundled with OMF
> > >
> > > The user has just downloaded and installed a package that comes with one
> > > or more bundled documents, and the package maintainer has thoughtfully
> > > provided an OMF file for each document.
> > >
> > > Each document has been provided in HTML and PDF format.
> > >
> > > Suppose the user installs the HTML documentation first, and installs it
> > > it into /usr/local/share/doc/app-name/html/, where normal file is called
> > > index.html.
> > >
> > > [ Everywhere I say "the user runs" I really mean "The package maintainer
> > > does this by means of a post-install script that they've already written.
> > > It is assumed that this post-install script knows the directories that
> > > documentation has been installed in, by virtue of being part of the
> > > package that did the installation in the first place. ]
> > >
> > > The user runs
> > >
> > > sk-install-doc -format html-split -db /var/db/sk/master.xml \
> > > /usr/local/share/doc/app-name/html/index.html
> >
> > Why do we specify the format on the command line? The FORMAT is specified
> > in the OMF already. The packager should just install and register the
> > metadata instead of putting it on the command line.
>
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we
> give it.
>
> This separation is necessary to support documents that have no separate
> OMF. Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.
ScrollKeeper is here because (and for) Document metadata - if it doesn't
exist, then ScrollKeeper should not know about the file.
[SNIP]
Too be honest, I got tired of reading this very very very long mail (it's been
one hour!!!!)
Anyway - I think somebody needs to come up with SPECIFIC suggestions, and
split this really long thread/mail into separate smaller, more manageable
threads.
Regards,
Ali
|
|
From: Dan M. <d-...@uc...> - 2000-11-24 07:44:51
|
On Thu, 23 Nov 2000, Ali Abdin wrote: > Too be honest, I got tired of reading this very very very long mail (it's been > one hour!!!!) > > Anyway - I think somebody needs to come up with SPECIFIC suggestions, and > split this really long thread/mail into separate smaller, more manageable > threads. Thanks Ali, Nik, and everybody else who has been reading and/or replying to these very long emails. I think it is a great idea to distill the biggest unresolved issues into seperate shorter threads. Dan |