Menu
▾
▴
Re: [Scrollkeeper-devel] Handling relocatable packages
|
From: Laszlo K. <las...@su...> - 2000-11-23 10:02:01
|
Guys,
You have to cut back on the email size, otherwise nobody is going to
contribute to this discussion.
Laszlo
>
> On Thu, 23 Nov 2000, Nik Clayton wrote:
>
> > On Tue, Nov 21, 2000 at 04:39:18PM -0600, Dan Mueth wrote:
> > > > There should be no such thing as a non-relocatable package.
> > >
> > > Both relocatable and non-relocatable packages exist, so scrollkeeper
> > > should support them both in the simplest possible way.
> >
> > Can you give me an example of non-relocatable package, and it's
> > justification for being non-relocatable?
>
> On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> gives me 506 packages which are non-relocateable, the last of which is
> xbill:)
>
> I think Red Hat would be better qualified at explaining this decision.
>
> > > The current way things are designed is that it will have one contents list
> > > for the system.
> >
> > I think that's a bad idea.
> >
> > Also, I should be clear here -- I'm not talking about how the system is
> > currently designed, or what the code currently does. I'm describing
> > what I think SK should do, and what the interfaces should be. It's kind
> > of hard to evaluate the existing functionality at the moment, because I
> > haven't seen a roadmap that lists explicit design decisions, and things
> > that will and won't be supported.
> >
> > > Of course you can always create another one somewhere
> > > else using the -p flag, but generally they would contain the same
> > > information and only the default one will get updated each time a document
> > > is installed. By specifying which OMF directories you want to include,
> > > these databases could be made to differ. We don't currently have a
> > > mechanism for supporting or specifying multiple OMF directories, but that
> > > would be fairly straightforward and is planned. This will make it very
> > > simple for users to have their own local databases which vary from the
> > > system's database by having OMF files in, say, $HOME/.scrollkeeper/omf
> > > which add to or over-ride the system defaults.
> >
> > This now sounds different from design doc #3, which, IIRC, suggested
> > that the contents list, and the OMF information, would be kept separate.
>
> I must have been unclear above. We definitely want to keep all of the
> original metadata in OMF files and seperate from the database. These OMF
> files should never be edited after they are initially installed (and
> possibly initially fixed to have the right IDENTIFIER during the
> post-install script, for relocatable packages). This way we are
> guaranteed to have an authoritative copy preserved in case the databases
> are corrupted and to make upgrades simpler.
>
> What I was trying to say above is that you can specify where the database
> lives with a -p <path> flag. If I run scrollkeeper-update -p /tmp/junk/
> and /tmp/junk/ doesn't exist or is empty, it creates the directory, reads
> all of the OMF files in the directory(ies) it is configured to read, and
> then dumps a brand new database(ie. contents list, toc, index, ...) under
> /tmp/junk/. This database won't generally be any different from the one in
> its normal location (/var/lib/scrollkeeper/) unless it is running under a
> different configuration. This would happen for a user who has his own
> configuration which specifies that OMF files in $HOME/.scrollkeeper/omf/
> should be added to the database (typically under $HOME/.scrollkeeper/db/
> perhaps).
>
> > > It is not clear how and why one would want to use multiple contents lists
> > > except for individual users and/or hand customization. The problem is
> > > that somebody has to specify what each database contains. So it works for
> > > users, because they can hand-tweak the contents. It doesn't work very
> > > well for system-wide databases, since everything must be specified by the
> > > packager. So while you could imagine GNOME documents being kept in one
> > > list and KDE documents being kept in another list, this doesn't really
> > > achieve anything.
> >
> > I think it does, as long as a mechanism is provided so that sk can work
> > out where to put the information if the user doesn't indicate otherwise.
> > A config file that specifies the default Contents List should suffice.
> >
> > > The information which would be passed on the command
> > > line to sk to indicate which list the doc goes in is really metadata which
> > > belongs in the OMF. OMF and XML provide everything we need to have one
> > > database and understand the scope of each piece. (eg. whether it is a
> > > GNOME doc or KDE doc.)
> >
> > If you support multiple Contents Lists (and you have to, just to support
> > end users installing docs and not having write access to the system wide
> > Contents List) then you need to be able to specify which Contents List
> > to use.
>
> We definitely want to support multiple databases, each of which has a
> Contents List.
>
> I am still uncertain if we want inheritance between contents lists though.
>
> In many cases users will not want inheritance, because they lose any
> control over what they inherit. If instead they generate their own
> database, they have complete control over all its parts, including those
> parts which they would have otherwise inherited. One way they could do
> this is to over-ride one or more OMF files by keeping local
> copies. Another way is to have a local categories file which differs from
> the system's. This approach gives the user complete control over their
> database (including their contents lists) which would be much more
> difficult and awkward to achieve by inheriting.
>
> Now we can come up with instances where inheritance is useful. If a user
> wanted all the man pages in their help system, they might be better served
> by inheriting the system man page database. They would be limited though,
> in that they could not specify which sections of the man pages they want.
> This would be solved by having a database for each section of man
> pages. Then these sections could be inherited by users which would be
> much faster than regenerating them for each user.
>
> I think we really need to be very clear on how this will work and how it
> will be used. Would all databases be generated with the same categories
> list? If not, how do you handle merging them? Presumably there is a
> "default" categories list which gets enforced on the final contents list.
>
> I think performance measurements would be useful here. The example above
> of using inherited databases with man pages has an advantage because
> updating the database is faster. It may save disk space, but this is
> probably negligable. It has the downside of slowing down run-time
> performance and making things substantially more complex and fragile.
>
> > Yes, you could put that information in the OMF, but, as I understand it,
> > the OMF contains meta information about the document. The location of
> > the Contents List is not meta information about the document, so should
> > not be in the OMF.
>
> If I understood you, the "system" (ie nothing under /home) would have
> multiple databases. So when documentation is registered, it must be
> determined which database to register it in. Since this must be passed to
> the registration script, it is determined by the packager based on
> information about the content of the docs. Thus, it is based on document
> metadata. (eg. this is a man page so it goes in the man page database,
> this one is a KDE doc so it goes in the KDE database, ...) This type of
> metadata is stored in the OMF file already.
>
> What various databases do you envision?
>
> [SNIP]
>
> > > What is the advantage of breaking the main database into pieces?
> >
> > Flexibility for the future.
> >
> > > > 2. Maintaining a TOC from the Contents List
> > > >
> > > > 3. Maintaining an Index from the Contents List
> > > >
> > > > 4. Maintaining a Categories list
> > >
> > > Well, this is somewhat debatable and worthy of discussion. Somebody has
> > > to provide and maintain the categories list. On one side, we can just
> > > define it as part of OMF/sk and install a static list with sk. Then no
> > > maintenance is necessary. Or, we may consider having this specified by a
> > > configuration variable (say in /etc/sk.conf or $HOME/.skrc). This way a
> > > sys admin could drop in a replacement if they want. For example you could
> > > chop out all the HOWTO's and man pages if you know newbies who never touch
> > > a command line will be using your machine.
> >
> > I don't think SK should mandate the existence of a categories list, or
> > mandate that certain categories should exist. I expect that some well
> > defined categories ("Section 1 man page", "FAQ", "Manual", "HOWTO",
> > "Gnome") will spring up.
>
> We could provide an option to not use categories lists. We definitely
> should provide the ability to use them however, and we should specify a
> suggested list.
>
> > However, that's not the same thing as saying
> >
> > If a categories file exists (in some, as yet undecided format), SK
> > should have the option of warning the SA that they're about to install
> > a document with a category that does not currently exist, and asking
> > whether they want to continue. Naturally, the SA should have the
> > option of overriding this check with a command line flag.
>
> We could provide a warning and even an interactive mode as an option.
>
> > > > [ Note: I've explicitly ommitted searching from this list -- AIUI, someother
> > > > application will have the responsibility for handling user queries of the
> > > > SK TOC and Index files ]
> > >
> > > sk should export searching functionality.
> >
> > How? libsk.so? By having a program that accepts search parameters and
> > returns XML with the results? Some other mechanism?
>
> We definitely want libsk.so. I'm not sure if we would want a command-line
> program for this.
>
> > > > Activities (2) and (3) require a Contents List. Note that a don't say "the
> > > > {TOC,Index}", I say "a {TOC,Index}". This is because sk may be asked to
> > > > maintain several different TOCs and Indices from several different Contents
> > > > Lists on the same host (for example, a regular user might want to run sk
> > > > to maintain their own index and TOC of ~/my-docs).
> > >
> > > For a given database, I agree. Eventually, I see the system having one
> > > database (with one contents list) and possibly each user having one. Is
> > > this what you have in mind? If you think we want more than one contents
> > > list in the system's database, please explain why.
> >
> > Flexibility and efficiency. Suppose you have one big contents list that
> > contains information about all the documentation that's installed on the
> > system -- man pages, readmes, faqs, and so on. That's going to be a
> > very big contents list, particularly when apps like Gnome and KDE get in
> > on the act.
>
> It will be big. Hopefully we can start running some tests on performance
> some time soon to see how big of an issue this becomes. It is not clear
> that merging two medium sized lists into a big list is any better than
> starting with a big list. You still have a big list in the end, only you
> save the part where you have to process and store two medium lists all the
> time.
>
> > For example, on a BSD system, the contents of /usr/share/man change once
> > in a blue moon, as third party manual pages generally live under
> > /usr/local/share/man. So the admin might want to create a Contents List
> > for /usr/share/man once, and keep it separate, because the SA knows its
> > never going to change, so it doesn't need to be included in any
> > processes SK undertakes to ensure that the Contents List is sync with
> > the filesystem.
>
> This would speed things up a bit when one installs a new OMF file if man
> pages were treated the same as other documents. However processing of man
> pages are done seperate from other applications since the OMF files for
> most (all?) man pages are generated and can be stored in a special
> location. This way the man pages are updated with cron each night (which
> must also make sure to create OMF metadata for each man page) and the
> normal docs are registered whenever an application installs. Because they
> are processed at different times they do not slow each other down.
>
> Aside from rebuilding the database in cases where it becomes corrupt, I'd
> like this to be a zero maintenance system that works out of the box. I
> don't think we should focus on unusual things a SA may want to
> do. Instead we should make it work well and in a robust manner so the SA
> won't be tempted to try to "fix" it.
>
> > > > What sort of documentation can the Contents List going to point to?
> > > >
> > > > 1. Locally installed standard documentation that comes with the system,
> > > > such as man pages, or Info documents.
> > >
> > > Perhaps. I think we probably do want to include this information, using
> > > automagically generated (and therefore incomplete) OMF metadata as you
> > > suggest below. For man and info, we would have to update this information
> > > off of cron unfortunately. So there will be an unavoidable delay between
> > > when a document is installed and when it is registered with sk.
> >
> > I don't quite agree. As I say below, I think installing a new document
> > should be a two step process, at least conceptually (as it could be
> > carried out by one command).
> >
> > First, the user tells SK about the documentation.
> >
> > Second, the user assigns some OMF to the documentation.
> >
> > This is what I'm getting at with the sk-install-doc and sk-install-omf
> > command separation.
>
> Why do we need the former step? The OMF data includes the IDENTIFIER,
> which makes the first step unnecessary.
>
> [SNIP]
>
> > Note that I don't mean that SK goes and writes a new, master contents
> > list file. Just that, in memory, SK is capable of aggregating
> > information from multiple Contents Lists (which should be accessible by
> > URL, so that remote Contents Lists are feasible).
>
> If we import a contents list it could only have documents on the Net on it
> or the docs would generally be unreachable. However this could be a very
> useful feature. You could imagine a given project publishing a database
> and docs on the Net and then importing them into your local
> system. Without the ability to inherit contents list one would have to
> install the metadata locally. This would be possible and could even be
> automated. We would need to consider this carefully to determine which
> solution is simpler and more robust.
>
> > > > DD: sk must have a run time option to allow the end user to select which
> > > > TOC to update.
> > >
> > > The databases are stored together. So, this should be specified at the
> > > same time as the contents list part of the database. I don't think I want
> > > my contents list coming from one database and my toc from another.
> > >
> > > (Or am I misunderstanding you? I'm not sure what you mean by "update" a
> > > TOC.)
> >
> > It's my understanding that there's a 1:1 mapping between a Contents List
> > and a ToC. That is, you can't have a ToC that includes information
> > about documents that aren't in the Contents List.
>
> Yes.
>
> > So if you have multiple Contents Lists, you need to have multiple ToCs
> > (and Indexes), and a way of saying that >this< Contents List was used to
> > produce >This< ToC.
>
> Not if only one contents list/toc pair live in each database directory.
>
> > Or are you saying that the ToC should be stored within the Contents List?
>
> No.
>
> [SNIP]
>
> > Do you agree that making the distinction between a document, and one or
> > more instances of that document (different formats, different languages)
> > is worthwhile.
>
> The only way scrollkeeper knows that two docs of different
> versions/formats/languages are otherwise the same is using the extra OMF
> element (which doesn't yet exist) which assigns them an ISBN-type code
> which is the same. Since they have different IDENTIFIERS (and other
> metadata), sk knows they are different documents.
>
> [SNIP]
>
> > > Why do we specify the format on the command line? The FORMAT is specified
> > > in the OMF already. The packager should just install and register the
> > > metadata instead of putting it on the command line.
> >
> > Because at this point, all we've done is install a document with no OMF.
> > SK has to synthesise some OMF, based on any additional information we
> > give it.
> >
> > This separation is necessary to support documents that have no separate
> > OMF. Adding "-format" to the command line is probably going to be
> > easier for an end user than having to create a separate file to return
> > to.
>
> Perhaps we agree on this after all and are just talking past each
> other. So for docs with OMF we have two steps:
> 1) update the IDENTIFIER to point to the right URL
> 2) register the OMF file with sk
>
> Really, #1 is only necessary for local docs (as opposed to remote
> ones) and for relocatable packages.
>
> For docs without OMF, we do two steps:
> 1) generate the best OMF file we can given the doc
> 2) register the OMF file with sk
>
> > > Here I assume (hope?;) that you mean the database directory and not the
> > > contents list. If so, we already do this with the "-p" flag.
> >
> > Right.
>
> So we would have multiple databases, each with exactly one Contents List,
> TOC directory, etc., right?
>
> > I'd like to see a mechanism to specify it on the command line (apart
> > from anything else, it's easier to script). sk-build-omf, as you
> > suggest above, seems like a good compromise.
>
> :)
>
> > > Right now we just number them, which guarantees a unique identifier.
> >
> > Two problems with this:
> >
> > 1. Probably won't scale efficiently as the Contents List gets larger.
> >
> > 2. Makes it impossible to merge two separate Contents Lists, from two
> > or more sites (say you're on a univesity campus, where the biology
> > department and the chemistry department both make available their
> > list of documentation in two different Scroll Keeper Contents Lists).
>
> When sk merges seperate databases, it can easily mark them so that it
> knows where the pieces came. One way to do this would be to append a few
> digit code to the beginning of each id which describes which database it
> came from.
>
> > Your document browser is going to need a mechanism to uniquely
> > identify each document. If there's a strong probability that the
> > document IDs are not unique then the browsers are each going to
> > have to reinvent their own mechanism for ensuring that the
> > documents are unique.
> >
> > The documents URL might be appropriate, but it's a bit unwieldy.
>
> The document URL is a very good unique identifier, and it is already being
> passed to the help browser.
>
> > You really need a mechanism that can (as near as possible) guarantee
> > globally unique document IDs.
>
> I'm pretty sure that only one document lives at a given URL ;)
>
> > > > As you can see, not a lot of information in there so far. "lang" and
> > > > "encoding" could also be specified on the command line, and if not
> > > > specified, some defaults are used.
> > > >
> > > > I think we probably also need a modification time attribute, but I haven't
> > > > shown that here.
> > >
> > > The modification time is stored in a seperate file which keeps track of
> > > which documents are in the database and is used for database maintenance,
> > > updating, installing, uninstalling, etc. The contents list does not care
> > > about the modification time.
> >
> > It probably should. Apart from the issues I outline above, this then
> > also lets you do things like "Show me the documents in order, most
> > recently modified first".
>
> This sort of filtering should be handled by sk, which knows the timestamps
> even though they are not part of the contents list. (The contents list is
> there to sort the documents according to a given set of instructions - the
> categories file. It is not the primary source for the metadata or
> metametadata. We have a seperate metametadata file.)
>
> > > > ==========================================================================
> > > > Category: Unfiled
> > > >
> > > > The App-Name manual
> > > >
> > > > Include meta information from the OMF here (SK ID = MD5)
> > > >
> > > > Formats: Local HTML (split), PDF, Remote HTML (split)
> > > > =========================================================================
> > >
> > > This may be useful for people who don't have a help-browser aside from
> > > Netscape. Although Netscape won't know how to display most docs.
> >
> > True. However, we should allow the Index builder flexibilit in deciding
> > what documents are included in the Index. This may include producing a
> > special Index that only contains documents available in HTML, plain
> > text, and PDF (for example).
>
> What do you mean by "index"? The definition I gave for "index" in prop#3
> is what you find in the back of a book - an alphabetical list of concepts
> and links to where they are discussed in the given work. My impression is
> that you mean the Contents List here. Is that right?
>
> I agree that we will want searching/filtering features in sk that allow
> people to control its behavior as you suggest.
>
> Dan
>
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/mailman/listinfo/scrollkeeper-devel
|