Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Ali A. <ali...@au...> - 2000-11-24 00:32:40
|
* Nik Clayton (ni...@no...) wrote at 19:17 on 23/11/00:
> On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > There should be no such thing as a non-relocatable package.
> >
> > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > should support them both in the simplest possible way.
>
> Can you give me an example of non-relocatable package, and it's
> justification for being non-relocatable?
GNOME is non-relocatable...Nautilus is non-relocatable (I maybe incorrect, somebody
told me they were non-relocatable)
The reason is the "datadirs" are define'd at compiletime.
Lets take an example....I have an application called gLife that uses a glade
file for its UI. The glade file is installed in $(prefix)/share/glife (prefix
is defined at ./configure time)....so to load the glade file, I need the
/path/ to the glade file. This is defined during compilation as GLIFE_DATADIR.
Now, if you compile the application into an RPM for '/usr' then you try
installing it in '/opt', the application will still look for the glade file in
/usr/share/glife.
Now, I know there are probably hacks that can be done to get around this. I
know that this may not be "theoretically" correct, etc. etc. etc. But the fact
is there /ARE/ non-relocatable packages out there, and scrollkeeper should
support both types of packages.
> > The current way things are designed is that it will have one contents list
> > for the system.
>
> I think that's a bad idea.
>
> Also, I should be clear here -- I'm not talking about how the system is
> currently designed, or what the code currently does. I'm describing
> what I think SK should do, and what the interfaces should be. It's kind
> of hard to evaluate the existing functionality at the moment, because I
> haven't seen a roadmap that lists explicit design decisions, and things
> that will and won't be supported.
>
> > Of course you can always create another one somewhere
> > else using the -p flag, but generally they would contain the same
> > information and only the default one will get updated each time a document
> > is installed. By specifying which OMF directories you want to include,
> > these databases could be made to differ. We don't currently have a
> > mechanism for supporting or specifying multiple OMF directories, but that
> > would be fairly straightforward and is planned. This will make it very
> > simple for users to have their own local databases which vary from the
> > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > which add to or over-ride the system defaults.
>
> This now sounds different from design doc #3, which, IIRC, suggested
> that the contents list, and the OMF information, would be kept separate.
I agree, they should be separate.
> > It is not clear how and why one would want to use multiple contents lists
> > except for individual users and/or hand customization. The problem is
> > that somebody has to specify what each database contains. So it works for
> > users, because they can hand-tweak the contents. It doesn't work very
> > well for system-wide databases, since everything must be specified by the
> > packager. So while you could imagine GNOME documents being kept in one
> > list and KDE documents being kept in another list, this doesn't really
> > achieve anything.
>
> I think it does, as long as a mechanism is provided so that sk can work
> out where to put the information if the user doesn't indicate otherwise.
> A config file that specifies the default Contents List should suffice.
>
> > The information which would be passed on the command
> > line to sk to indicate which list the doc goes in is really metadata which
> > belongs in the OMF. OMF and XML provide everything we need to have one
> > database and understand the scope of each piece. (eg. whether it is a
> > GNOME doc or KDE doc.)
>
> If you support multiple Contents Lists (and you have to, just to support
> end users installing docs and not having write access to the system wide
> Contents List) then you need to be able to specify which Contents List
> to use.
>
> Yes, you could put that information in the OMF, but, as I understand it,
> the OMF contains meta information about the document. The location of
> the Contents List is not meta information about the document, so should
> not be in the OMF.
>
> Also, consider a regular user who occasionally installs applications and
> documentation in to $HOME.
>
> It's much easier for them to add a flag to a command line (or, better
> still, set $SCROLLKEEPER_DEFAULT_CONTENTS in their shell start up files)
> than it is for them to edit OMF files before installing them.
>
> > What is the advantage of breaking the main database into pieces?
>
> Flexibility for the future.
There should be support in mind for "multiple" databases in my opinion. This
way, a person can set an environment flag, or a command-line option and be
able to load an entirely new database (along with his old one).
Perhaps you should have a '/etc/scrollkeeper.conf' file that has a list of
various databases as well as various configuration options.
In addition to this, I think scrollkeeper should handle any databases found in
'~/.scrollkeeper/'
So, multiple databases does make sense in my opinion (think of it this way,
just because YOU don't need it, or can't find a reason to use it, it doesn't
mean that it is not useful for others)..
From a desktop perspective it might not be necessary to have multiple
databases, but from a "sysadmin" perspective it does.
> > > 2. Maintaining a TOC from the Contents List
> > >
> > > 3. Maintaining an Index from the Contents List
> > >
> > > 4. Maintaining a Categories list
> >
> > Well, this is somewhat debatable and worthy of discussion. Somebody has
> > to provide and maintain the categories list. On one side, we can just
> > define it as part of OMF/sk and install a static list with sk. Then no
> > maintenance is necessary. Or, we may consider having this specified by a
> > configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> > sys admin could drop in a replacement if they want. For example you could
> > chop out all the HOWTO's and man pages if you know newbies who never touch
> > a command line will be using your machine.
>
> I don't think SK should mandate the existence of a categories list, or
> mandate that certain categories should exist. I expect that some well
> defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> "Gnome") will spring up.
FAQ/Manual/HOWTO are not category types in your opinion...What I mean by
categories is more like "subject" - i.e. "Audio/MP3 Players' or
'Office/Spreadsheet' or 'X11/Internet/FTP Clients' etc.
The fact that it is a FAQ/HOWTO/Manual should be reflected in the name of the
document (I think this is the "purpose" of a document).
There should also be another field that specifies the type of document (HTML,
Info, Man, DocBook, plain text, etc.)
> However, that's not the same thing as saying
>
> If a categories file exists (in some, as yet undecided format), SK
> should have the option of warning the SA that they're about to install
> a document with a category that does not currently exist, and asking
> whether they want to continue. Naturally, the SA should have the
> option of overriding this check with a command line flag.
I am saying just a HTML file, or plain text file just LISTING the categories -
nothing fancy...Scrollkeeper would have no knowledge of this file and will not
interact with it in anyway.
The scrips should NEVER EVER become "interactive". Just use whatever is in the
category (in the OMF file).
> > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > application will have the responsibility for handling user queries of the
> > > SK TOC and Index files ]
> >
> > sk should export searching functionality.
>
> How? libsk.so? By having a program that accepts search parameters and
> returns XML with the results? Some other mechanism?
I prefer a library that returns the xmlDocPtr of the file...you can then parse
it as you wish. Note: with a library you could _EASILY_ create a command-line
application that would just "dump" the XML tree to stdout (if you need it)
> > > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > > maintain several different TOCs and Indices from several different Contents
> > > Lists on the same host (for example, a regular user might want to run sk
> > > to maintain their own index and TOC of ~/my-docs).
> >
> > For a given database, I agree. Eventually, I see the system having one
> > database (with one contents list) and possibly each user having one. Is
> > this what you have in mind? If you think we want more than one contents
> > list in the system's database, please explain why.
>
> Flexibility and efficiency. Suppose you have one big contents list that
> contains information about all the documentation that's installed on the
> system -- man pages, readmes, faqs, and so on. That's going to be a
> very big contents list, particularly when apps like Gnome and KDE get in
> on the act.
>
> For example, on a BSD system, the contents of /usr/share/man change once
> in a blue moon, as third party manual pages generally live under
> /usr/local/share/man. So the admin might want to create a Contents List
> for /usr/share/man once, and keep it separate, because the SA knows its
> never going to change, so it doesn't need to be included in any
> processes SK undertakes to ensure that the Contents List is sync with
> the filesystem.
I think this "optimization" is unncessary at this time. We have no data _AT
ALL_ on the efficiency in scrollkeeper. We don't know if it will be slow or
fast, so splitting up databases for this reason is unacceptable in my opinion.
On the other hand, I do think we should support multiple databases (through a
command-line option, and possible an environment variable (and API functions
when the library is done))
> > > What sort of documentation can the Contents List going to point to?
> > >
> > > 1. Locally installed standard documentation that comes with the system,
> > > such as man pages, or Info documents.
> >
> > Perhaps. I think we probably do want to include this information, using
> > automagically generated (and therefore incomplete) OMF metadata as you
> > suggest below. For man and info, we would have to update this information
> > off of cron unfortunately. So there will be an unavoidable delay between
> > when a document is installed and when it is registered with sk.
>
> I don't quite agree. As I say below, I think installing a new document
> should be a two step process, at least conceptually (as it could be
> carried out by one command).
>
> First, the user tells SK about the documentation.
>
> Second, the user assigns some OMF to the documentation.
>
> This is what I'm getting at with the sk-install-doc and sk-install-omf
> command separation.
>
> Consider the man page, for example. If I want to tell SK about all my
> section 1 man pages, I'd like to be able to do something like
>
> cd /usr/share/man/man1
> sk-install-doc *.1
>
> which would tell SK about all my section 1 manual pages. Actually, what
> I'd probably do is something like
>
> sk-install-doc -category "Man Pages:Section 1" *.1
>
> to assign a category to them. At this point, there's no OMF for them,
> but SK can synthesise the bare bones from the filename and the category.
>
> Suppose, then, that I want to add some OMF for ls.1. I might do
> something like this;
>
> sk-extract-omf ls.1 > ls.omf
> vi ls.omf
> [edit edit edit]
> sk-install-omf ls.1 ls.omf
>
> which would pull out the OMF that SK synthesised for me in the earlier
> step (presumably as XML). I can then edit it, and then install the
> edited OMF back in to SK's control.
>
> I think it's highly likely that *roff -man (and in particular, *roff
> -mdoc) is going to be easy to extract OMF from. So what we might end up
> with is a command to extract OMF from -man formatted documents. It
> might work like this;
>
> cd /usr/share/man/man1
> foreach man in (*.1)
> extract-omf -source man $man > `basename $man .1`.omf
> end
>
> which would iterate over *.1, extracting the OMF and writing it to
> *.omf. With that done, the end user might then be able to do
>
> foreach man in (*.1)
> sk-install-doc -doc $man -omf `basename $man .1`.omf
> end
>
> which could install the document, and the pregenerated OMF, into SK with
> one command.
>
> I hope the design ethos that's coming across from all of this is to have
> lots of small tools, each of which does one thing.
I disagree here...Without the OMF data ScrollKeeper shouldn't do jack shit to
the document. It shouldn't even know the document exists.
If somebody wishes to write an application that would take a document, create
an OMF file from the document, and register it with ScrollKeeper - well, that
would be nice. But its not /ScrollKeepers/ responsibility.
Maybe something like this can be done in the future, but I don't see this as
an 'essential feature'...
Also it would be "nice" to have a command-line tool that would allow you
modify the OMF metadata based on the command-line options.
> > > 2. Locally installed system documentation, installed by the SA team.
> >
> > SA =? sys admin
>
> Yes.
>
> > > 3. Locally installed documentation, installed by the user.
> >
> > Yes. Although like much of this discussion, this feature will come in a
> > post-0.1 release.
>
> As I say, I'm not thinking in terms of 'release' yet. I'd like to get
> some specific functionality nailed down and agreed on.
>
> [ Incidentally, and talking about the implementation -- one of the
> things that occured to me last night; sk could have a command
> structure like that of cvs, where instead of having a bunch of
> sk-foo-bar commands, you have one 'sk' command, that takes options,
> and then subcommands, which also take options.
>
> sk --contents /var/db/sk/contents.xml install-doc /path/to/doc
> sk install-omf /path/to/omf
>
> and so on. ]
Difficult I think since the SK stuff are two distinct separate tools :) but it
would be an interesting thing to do...
> > > I think it is fair to say that, when run, sk will need to construct it's
> > > global Contents List by parsing multiple, smaller, contents lists. In much
> > > the same way that SGML catalog files can include one another at the moment.
> >
> > Why? It sounds very nice and wonderful. However, this adds a layer of
> > complexity without any obvious benefits.
>
> As above.
>
> Note that I don't mean that SK goes and writes a new, master contents
> list file. Just that, in memory, SK is capable of aggregating
> information from multiple Contents Lists (which should be accessible by
> URL, so that remote Contents Lists are feasible).
You could just re-construct one large XML tree in memory from multiple smaller
XML files. You would then conduct all operations on this in-memory XML tree.
It won't make much diff. performance wise if you do this on one LARGE xml
file, or multiple smaller ones i believe (provided you are doing the above
approach of building a large XML tree from smaller ones).
> > > DD: sk must have a run time option to allow the end user to select which
> > > TOC to update.
> >
> > The databases are stored together. So, this should be specified at the
> > same time as the contents list part of the database. I don't think I want
> > my contents list coming from one database and my toc from another.
> >
> > (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> > TOC.)
>
> It's my understanding that there's a 1:1 mapping between a Contents List
> and a ToC. That is, you can't have a ToC that includes information
> about documents that aren't in the Contents List.
>
> So if you have multiple Contents Lists, you need to have multiple ToCs
> (and Indexes), and a way of saying that >this< Contents List was used to
> produce >This< ToC.
>
> Or are you saying that the ToC should be stored within the Contents List?
>
> > > DD: sk must have a run time option to allow the end user to select which
> > > Index to update.
> >
> > (as above)
>
> Ditto :-)
>
> > > DD: One of the sk commands must be to generate a usable Contents List
> > > (which really means {X}HTML version, to start with) given a Contents
> > > List which is in sk internal format.
> >
> > Everything is XML right now, so this is not an issue. If we decide
> > another internal format is better we should still export XML to the
> > browser. As for HTML, this is possible and may be a nice interface for
> > people who want to create a very minimalistic web browser.
>
> All the world is not yet XML, and not everyone runs KDE or Gnome. Being
> able to convert the ToCs and Indices to other formats is vital.
> Fortunately, since the ToC and Indices are XML, this should be easy,
> using something like XSLT, so we can probably just consider this
> functionality that we get for free.
Well, even if you don't use KDE or GNOME, you would still need to parse
ScrollKeeper's file-format...
Lets say we /don't/ use XML, you would still have to handle whatever internal
data structure is used by ScrollKeeper. We just picked XML because it is
(relatively) easier to parse.
> > > DD: It is possible for a document to change, but for the OMF information
> > > to be neglected. sk should maintain a modification time stamp for
> > > each document and its corresponding OMF, to allow the end user to
> > > determine when that might have happened, and to correct for it.
> >
> > For the current version of sk, which does not yet allow for remote
> > documents, only the OMF file's timestamp is watched. This is because if
> > somebody edits the doc and not the OMF, then it is their fault for not
> > updating the OMF, not sk's. In reality whenever you install a new version
> > of a package, even if the OMF is not updated (as it should), it still
> > updates the old version so it gets a new timestamp and the new doc gets
> > registered despite any negligence on the part of the maintainer to update
> > the OMF (or the doc for that matter;).
> >
> > This mainly becomes an issue for:
> > 1) remote docs
> > 2) docs which users write and modify themselves.
> >
> > So we may indeed need to start tracking the doc timestamps when we support
> > these features. We have to be careful about constantly checking the
> > timestamps on a bunch of docs which are spread around the internet
> > though. There will be many cases where this task cannot complete, or else
> > takes an unacceptable amount of time.
>
> This should be configurable behaviour, something like;
>
> sk --check-remote-timestamps --remote-timeout=30
>
> Actually, this implies another DD
>
> DD: sk must track the validity of the OMF in relation to it's document.
> OMF might be:
>
> valid This OMF was installed with the document, and neither
> the document or the OMF's timestamps have changed.
>
> invalid The document and OMF have different timestamps
>
> unknown The document and OMF have the same timestamps, but
> the most recent attempt to confirm this (particularly
> for remote documents) failed.
>
> sk should be able to report on documents who have OMF which is not
> valid, so the SA can investigate further.
>
> > > DD: One document, multiple output formats. The user might have installed
> > > the same document three times, once as HTML, once as PDF, and once as
> > > plain text. The three installations are the same document, and have
> > > the same OMF associated with them.
> >
> > No. These definitely do not have the same OMF, unless they are identical
> > copies. The OMF file specifies the document format in FORMAT.DTD and
> > FORMAT.MIME. The docs also have different IDENTIFIER's.
> >
> > (We may decide to expand this a little bit. FORMAT.STYLESHEET may be
> > nice, as it would allow the doc authors to dictate the appearance of their
> > doc. So GNOME docs have a little GNOME icon, KDE docs have a little KDE
> > icon, etc.)
> >
> > > sk should treat this as one document with three formats, rather than
> > > three separate documents.
> >
> > If you look in the ldp-meta archives from several months ago, I proposed
> > we add a document id to the OMF. This would allow us to identify if two
> > documents are "The-Same". I call two documents "The-Same" if they are the
> > same document only different versions, languages, formats, etc. If two
> > documents have the same title, they are not necessarily
> > "The-Same". Likewise, if I change the title of my document at some point
> > in time, I can have two docs which are "The-Same" but not having the same
> > title.
> >
> > Obviously this sort of information will be very nice to have. You can do
> > things like "Get me this document in language xxx." or "Show me the latest
> > version of this document on the Net." or "Show me the previous version of
> > this document from the Net."
> >
> > > [ This is probably going to be the most contentious piece of this
> > > writeup, and it might change. In particular, it also suggests that
> > > translations of a document should be treated as the same document,
> > > just in a different language, and I'm not sure that's a good idea.
> > > Then again, it might be. ]
> >
> > I guess we were thinking along similar lines :)
>
> Do you agree that making the distinction between a document, and one or
> more instances of that document (different formats, different languages)
> is worthwhile.
Translations are considered to be "separate" documents for me.
I don't know about different formats...that is a difficult one - but why would
we need to distinguish between instances of a document based on format?
I like the idea of an "id" of a document, because then you can have diff.
versions of a doc. It is difficult to determine wether translations should
have their own id...I recommend you just flip a coin.
The good thing about an id (unique id's can be obtained using 'uuid-gen') is
that you can query any server in the globe for that particular id and it will
retain that document (provided you also provide the version number, and
possible the format)
> > > There are a number of tasks that the sk end user is going to want to tell
> > > sk to do. They are;
> > >
> > > Install new documentation that comes bundled with OMF
> > >
> > > The user has just downloaded and installed a package that comes with one
> > > or more bundled documents, and the package maintainer has thoughtfully
> > > provided an OMF file for each document.
> > >
> > > Each document has been provided in HTML and PDF format.
> > >
> > > Suppose the user installs the HTML documentation first, and installs it
> > > it into /usr/local/share/doc/app-name/html/, where normal file is called
> > > index.html.
> > >
> > > [ Everywhere I say "the user runs" I really mean "The package maintainer
> > > does this by means of a post-install script that they've already written.
> > > It is assumed that this post-install script knows the directories that
> > > documentation has been installed in, by virtue of being part of the
> > > package that did the installation in the first place. ]
> > >
> > > The user runs
> > >
> > > sk-install-doc -format html-split -db /var/db/sk/master.xml \
> > > /usr/local/share/doc/app-name/html/index.html
> >
> > Why do we specify the format on the command line? The FORMAT is specified
> > in the OMF already. The packager should just install and register the
> > metadata instead of putting it on the command line.
>
> Because at this point, all we've done is install a document with no OMF.
> SK has to synthesise some OMF, based on any additional information we
> give it.
>
> This separation is necessary to support documents that have no separate
> OMF. Adding "-format" to the command line is probably going to be
> easier for an end user than having to create a separate file to return
> to.
ScrollKeeper is here because (and for) Document metadata - if it doesn't
exist, then ScrollKeeper should not know about the file.
[SNIP]
Too be honest, I got tired of reading this very very very long mail (it's been
one hour!!!!)
Anyway - I think somebody needs to come up with SPECIFIC suggestions, and
split this really long thread/mail into separate smaller, more manageable
threads.
Regards,
Ali
|