scrollkeeper-devel — Developer discussion list for ScrollKeeper

Re: [Scrollkeeper-devel] Simple URI's

[Ali A. <ALI...@au...>]

On Tue, 28 Nov 2000, Dan Mueth wrote:

> 
> GNOME uses a "gnome-help" URI scheme.  So if you type
> "gnome-help:gnumeric", it looks in
> $prefix/share/gnome/help/gnumeric/<locale>/ and looks for the doc under
> the name gnumeric.sgml or else index.html.  It is a very convenient way to
> find a manual.  It also allows for cross-referencing of documents very
> simply since any document you write knows that "gnome-help:gnumeric" will
> point to the Gnumeric manual regardless of where things are installed (eg.
> under /usr or /opt or ...).
> 
> The problem with this is:
> 1) other help browsers have to understand how to handle the
> "gnome-help" URI scheme
> 2) presumably other desktops will introduce more URI schemes
> 3) many docs won't be addressable at all this way unless scrollkeeper
> provides a mechanism.
> 
> So probably what we want is to introduce a scheme which passes the
> identifier (what is the right term here?) to scrollkeeper which can return
> the path.  So for example, one would use: "sk-help:modem-HOWTO".  It
> would have to find the doc (and select from any available versions,
> languages, formats) and then display it.  This would be *very* useful IMO,
> both for users as well as for cross-referencing between documents.

This is a good idea
 
> A couple issues which pop up:
> 
> 1) Selecting the right doc.  We could have a default setting for language
> (just use the current locale), formats (perhaps in /etc/sk.conf we list in
> order of preference), and versions (show the biggest version number).

Use the current locale for "default", then fallback to the 'C' locale.

I think you also need to do some i18n handling (i.e. LANG can be a 
delimited list of locales (or is that LC_ALL?) (e.g. export 
LANG=fr_BR:fr_FR:en_GB:en_US)

> 2) Mapping the names.  The OMF file should specify the name by which the
> document should be referred.  This seems to require yet another OMF
> attribute to be created, perhaps IDENTIFIER.URINAME?  We will not have
> complete uniqueness here.  However ideally we will have a 1-1
> correspondence between NUID's and URINAME's.  We could keep a database on
> the web so people can register names and find out if a name is taken.  
> This creates even more administrative overhead, but I think it is well
> worth the trouble it would take to set up a simple php/mysql site on
> SourceForge.

Not sure this is a good idea...you will have to have a 'URINAME' if you 
are typing the URI manually.

BUT:

I think that applications should give scrollkeeper a "title", locale 
(optional), and version # (optional), format (default would be Docbook). 
Scrollkeeper would then search its database and return the unique IDs 
(possible there will be more than one doc matching the criteria, so the 
client would either pick the first one, or try to do some other 
processing (e.g: See Also)).

You could also have another function that would return the "first found 
match" of the document directly (so you don't have to search for UUID, 
and then tell scrollkeeper to return this document - you just tell it 
"return the first document that matches this criteria").

Anyway, this approach would only work if you have a list of documents to 
choose from. 

So, maybe URINAME's are the way to go.

[Scrollkeeper-devel] Simple URI's

[Dan M. <d-...@uc...>]

GNOME uses a "gnome-help" URI scheme.  So if you type
"gnome-help:gnumeric", it looks in
$prefix/share/gnome/help/gnumeric/<locale>/ and looks for the doc under
the name gnumeric.sgml or else index.html.  It is a very convenient way to
find a manual.  It also allows for cross-referencing of documents very
simply since any document you write knows that "gnome-help:gnumeric" will
point to the Gnumeric manual regardless of where things are installed (eg.
under /usr or /opt or ...).

The problem with this is:
1) other help browsers have to understand how to handle the
"gnome-help" URI scheme
2) presumably other desktops will introduce more URI schemes
3) many docs won't be addressable at all this way unless scrollkeeper
provides a mechanism.

So probably what we want is to introduce a scheme which passes the
identifier (what is the right term here?) to scrollkeeper which can return
the path.  So for example, one would use: "sk-help:modem-HOWTO".  It
would have to find the doc (and select from any available versions,
languages, formats) and then display it.  This would be *very* useful IMO,
both for users as well as for cross-referencing between documents.

A couple issues which pop up:

1) Selecting the right doc.  We could have a default setting for language
(just use the current locale), formats (perhaps in /etc/sk.conf we list in
order of preference), and versions (show the biggest version number).

2) Mapping the names.  The OMF file should specify the name by which the
document should be referred.  This seems to require yet another OMF
attribute to be created, perhaps IDENTIFIER.URINAME?  We will not have
complete uniqueness here.  However ideally we will have a 1-1
correspondence between NUID's and URINAME's.  We could keep a database on
the web so people can register names and find out if a name is taken.  
This creates even more administrative overhead, but I think it is well
worth the trouble it would take to set up a simple php/mysql site on
SourceForge.

Thoughts?

Dan

Re: [Scrollkeeper-devel] Category Trees

[Ali A. <ali...@au...>]

* Laszlo Kovacs (las...@su...) wrote at 17:14 on 27/11/00:
> > One other issue which should be discussed for options #1-#3 - What do we
> > do with empty parts of the tree?  Do we show branches that don't have
> > documents in them?  Or are these branches only created as they get
> > populated?   I think the assumption right now is that empty branches
> > exist and are passed to the browser.  (The browser could still potentially
> > snip them out.)
> 
> I don't think this is important from architectural point of view at the
> moment. This is why we chopped the big email to small emails and now we
> are growing it again:-)

I think the "core hackers" (you and Dan) should be more focused to on taking
decisions (even if they are bad ones) rather than having "endless"
discussions. Almost every point brought up has multiple approaches and each
point has pros and cons.

Maybe its just poor communication, but I haven't seen any decisions made (just
discussed to death).

Maybe that'll help the long emails ;)

Regards,
Ali

[Scrollkeeper-devel] draft article, for review

[Rich M. <rd...@cf...>]

I have written up some recent experiences with FreeBSD documentation
for my column (Silicon Carny) in SunWorld (http://www.sunworld.com).
The column also attempts to summarize the aims of the ScrollKeeper
and Meta Projects.

If you would like to look at the draft I'm submitting, take a look at
http://www.cfcl.com/rdm/SC.200012.1.  Please send me email (in the
next few days) about any problems you notice.  I should be able to
squeeze in a few changes during final edit...

-r
-- 
--
Rich Morin:          rd...@cf..., +1 650-873-7841, http://www.cfcl.com/rdm
Prime Time Freeware: in...@pt..., +1 408-433-9662, http://www.ptf.com
MacPerl: http://www.macperl.com, http://www.ptf.com/ptf/products/MPPE
MkLinux: http://www.mklinux.org, http://www.ptf.com/ptf/products/MKLP

Re: [Scrollkeeper-devel] Unique identifiers

[Dan M. <d-...@uc...>]

On Mon, 27 Nov 2000, Laszlo Kovacs wrote:

> Hi All,
> 
> During the category tree discussion the issue of unique identifiers for
> docs was raised. As we want to keep the various discssions as focused as
> possible I decided to start this thread.
> 
> Here are the main issues raised:
> 
> (Rich Morin)
> 	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4716298/
>         Each package (and each variant and version of same)
>         should have a unique identifier, so that we can be
>         clear and unambiguous in referring to it.  This is
>         really the same issue as the database folks have... 
> 
> (Rich Morin)
> 	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4716421/
> 	Use mnemonic identifiers such as package name and principal developer:
> 	e.g:	perl-lw
> 
> (Ali Abdin)
> 	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4718805/
> 	Use uuid-gen.
> 
> (Rich Morin)
> 	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4719633/
> 	uuid-gen is not available on every platform (notably FreeBSD).
> 
> We talked about this issue previously and we thought about an ISBN like
> ID generated by a server available throughout the internet. This server
> would generate unique IDs on request by document creators.

Alan and Daniel were kind enough to pop in on IRC to discuss scrollkeeper
yesterday and the conversation quickly moved to UUID's.  They recommended
(as Ali does) creating a UUID for each unique document.  Here a unique
document is probably a {doc, version, language, format} quadruplet, or
possibly even a superset of this.  It should uniquely specify the
resource.  We will need to decide where this belongs in the OMF, perhaps
an attribute to IDENTIFIER (ie. IDENTIFIER.UUID).  The UUID will be the
database key.  To make things simple, we can ship a script which generates
a UUID with scrollkeeper.  For systems with uuidgen, it will use that.  
For other systems, we may want it to request one from a server which runs
uuidgen (as suggested by Ali).  

Having this UUID system will keep things simpler when we implement server
catalogs and share docs/metadata between machines.

I'm worried that trying to generate UUID's by hand using some system as
suggested by Rich will make things more complicated and less unique.

Just for completeness, I'll mention that we also want another ID number
which we associate with each document.  We need a name - perhaps "docid"
or "nuid" (non-universally unique id).  Multiple versions, languages,
formats, etc. of the same document should have the same NUID.  This allows
us to do things like look for a given document in another language, look
for a newer (or older) version of a given document, and look for a
different format of a document (like PS so we can print it). Perhaps we
could call this IDENTIFIER.NUID in the OMF.

Dan

[Scrollkeeper-devel] Unique identifiers

[Laszlo K. <las...@su...>]

Hi All,

During the category tree discussion the issue of unique identifiers for
docs was raised. As we want to keep the various discssions as focused as
possible I decided to start this thread.

Here are the main issues raised:

(Rich Morin)
	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4716298/
        Each package (and each variant and version of same)
        should have a unique identifier, so that we can be
        clear and unambiguous in referring to it.  This is
        really the same issue as the database folks have... 

(Rich Morin)
	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4716421/
	Use mnemonic identifiers such as package name and principal developer:
	e.g:	perl-lw

(Ali Abdin)
	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4718805/
	Use uuid-gen.

(Rich Morin)
	http://www.geocrawler.com/lists/3/SourceForge/6429/0/4719633/
	uuid-gen is not available on every platform (notably FreeBSD).

We talked about this issue previously and we thought about an ISBN like
ID generated by a server available throughout the internet. This server
would generate unique IDs on request by document creators.

Anybody has any other ideas?

Laszlo

Re: [Scrollkeeper-devel] Category Trees

[Laszlo K. <las...@su...>]

>    *  unique identifiers
> 
>         Each package (and each variant and version of same)
>         should have a unique identifier, so that we can be
>         clear and unambiguous in referring to it.  This is
>         really the same issue as the database folks have...
I think this is a separate issue I might start up a new thread about it.

Laszlo

Re: [Scrollkeeper-devel] Category Trees

[Laszlo K. <las...@su...>]

[SNIP]
> 3) Typos and variations on translations.  Unless we do #2, we will always
> get spurious duplicated branches from these.
We would probably have to create/recommend categories for every
available locale consistently. I mean we should translate the categories
to every locale and promote these categories for the appropriate
locales.

> The difference between this and #3 is that #3 will have empty branches.
> I haven't decided whether I think this is a good thing or a bad thing.
> If the tree on the web page is huge and has large sections which are
> frequently empty, it may be best not to have them.
In case we go with a predefined category tree we will have empty
branches. Ignoring them in an XML environment should be trivial and left
to the browser.

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Ali A. <ali...@au...>]

* Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
> On Wed, Nov 22, 2000 at 09:55:45PM -0600, Dan Mueth wrote:
> > > Can you give me an example of non-relocatable package, and it's
> > > justification for being non-relocatable?
> > 
> > On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> > gives me 506 packages which are non-relocateable, the last of which is
> > xbill:)
> > 
> > I think Red Hat would be better qualified at explaining this decision.
> 
> Someone else bought up Gnome as an example of non-relocatable package.
> 
> Just because the original implementors didn't provide a hook to make it
> non-relocatable, don't assume that other's won't have done.

This is not scrollkeeper's problem though. We should support both instead of
imposing relocatable packages on people.

Actually, I think if we supposed relocatable packages, we will (by definition)
have supported non-relocatable packages, so this is a moot point.
 
> > I am still uncertain if we want inheritance between contents lists though.
> > 
> > In many cases users will not want inheritance, because they lose any
> > control over what they inherit.  
> 
> "many cases" means, therefore, that there will be times when users will
> want inheritence.
> 
> There's a saying that's prevalent in the BSD world; "mkae tools, not policy".
> Having read some of the later messages in this thread, I'm going to be
> coming back to that quite a lot.
> 
> I don't care what I, personally, might need some sk.  However, it's
> important not to impose artificial limitations that impose my (or your)
> policy for how to configure a host/network on someone else.

I don't see how scrollkeeper is imposing policy. We want it to support both
relocatable and non-relocatable packages. 

It seems that you want to to "impose" the policy of relocatable packages only
(although the concept of relocatable packages is policy imposing).
 
> > We definitely want libsk.so.  I'm not sure if we would want a command-line
> > program for this.
> 
> It's not something we need (if someone else needs it then they can write 
> it).  It'll probably prove to be a useful small sample application
> though.

Agreed. This shouldn't be one of the "targets" for 1.0, but if somebody does
it we could include it. Also, if post-1.0 comes and it is deemed a
command-line tool is needed, then we can worry about it

> <snip>
> 
> > Aside from rebuilding the database in cases where it becomes corrupt, I'd
> > like this to be a zero maintenance system that works out of the box.  I
> > don't think we should focus on unusual things a SA may want to
> > do.  
> 
> "Tools, not policy" (see, I said I'd be coming back to it).  The things
> that you are I think are unusual might be standard practice for someone
> who wants to use sk.

I think the goal is to make scrollkeeper "automated" enough so that Desktop
users can use it (remember - Help Documents are (usually) for new users).

Now if a sysadmin wanted to play around and mess with ScrollKeeper, I think
this also should be possible. Now, what kind of things would a sysadmin want
to do that should be supported?

> > > So if you have multiple Contents Lists, you need to have multiple ToCs
> > > (and Indexes), and a way of saying that >this< Contents List was used to
> > > produce >This< ToC.
> > 
> > Not if only one contents list/toc pair live in each database directory.
> 
> Tools, not policy.
> 
> I might want to put the Contents List in one directory (which is
> private, and not available to the world), and publish a ToC.
> 
> We can certainly say
> 
>    If you don't tell it otherwise, sk will generate the ToC in the same
>    place as the Contents List.
> 
> but we should make it possible for the SA to change this behaviour if
> necessary.

I really don't think this is a major issue (to specify which directory a file
goes into). 

This is one of the minor details that can be taken care of.

The issue here is: support multiple ToCs and Contents Lists.

> > > Do you agree that making the distinction between a document, and one or
> > > more instances of that document (different formats, different languages)
> > > is worthwhile.
> > 
> > The only way scrollkeeper knows that two docs of different
> > versions/formats/languages are otherwise the same is using the extra OMF
> > element (which doesn't yet exist) which assigns them an ISBN-type code
> > which is the same.  Since they have different IDENTIFIERS (and other
> > metadata), sk knows they are different documents.
> 
> This is (possibly) something that can be handled outside of sk with
> creative use of the RELATION element in the OMF.

I've actually never seen an OMF file *goes and hangs his head in shame*

> > > > Why do we specify the format on the command line?  The FORMAT is specified
> > > > in the OMF already.  The packager should just install and register the
> > > > metadata instead of putting it on the command line.
> > > 
> > > Because at this point, all we've done is install a document with no OMF.
> > > SK has to synthesise some OMF, based on any additional information we 
> > > give it.
> > > 
> > > This separation is necessary to support documents that have no separate
> > > OMF.  Adding "-format" to the command line is probably going to be
> > > easier for an end user than having to create a separate file to return
> > > to.
> > 
> > Perhaps we agree on this after all and are just talking past each
> > other.  So for docs with OMF we have two steps:
> > 1) update the IDENTIFIER to point to the right URL
> > 2) register the OMF file with sk
> > 
> > Really, #1 is only necessary for local docs (as opposed to remote
> > ones) and for relocatable packages.
> 
> I think that's right.  In my example, "sk-install-doc" is probably the
> wrong name.
> 
> Ah ha, I think I've got a work around.
> 
> We've been going back and forth about what to do if a document doesn't
> have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
> wrong, or whatever.
> 
> So, what if we have two command, sk-generate-omf and sk-install-omf?  The 
> usage for this would look something like;
> 
>   sk-generate-omf [--omf:creator ...]
> 	  	  [--omf:maintainer ...]
> 		  [--omf:contributor ...]
> 		  [--omf:title ...]
> 		  [--omf:date ...]
>                   [<other --omf:... options]
> 		  file.omf
> 
>   sk-install-omf [--omf-dir ...] /path/to/omf/file
> 
> sk-generate-omf is used to generate OMF for installation into sk.  It can be 
> told about an existing OMF file (for example, if the document has been 
> distributed with an OMF file).  However, it also has a series of --omf:<foo>
> options, which are used to override the OMF in the .omf file, and replace 
> it with something else (I haven't listed them all in the synopsis above).

Umm - okay, lets say I have a well-formed OMF file and then I do
'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?

Hmm, you seem to be implying that people will not write OMF files and that
they should be "extracted" from the docs. From my understanding, they are a
part of the doc (there was discussion on wether to include necessary tags in
the docs themselves, or to have separate OMF files).

I think perhaps we should just allow the '--omf:' overrides in the
sk-install-omf binary

> sk-install-omf is used to install an OMF file into SK.
> 
> In the simplest case (where the document already has a separate OMF
> file), in the installation Makefile, this might look something like
> 
>     PREFIX=		/usr/local
>     APP=		appname
>     SK_OMF_DIR=		`sk-config --omf-dir`
> 
>     ...
> 
>     install-doc:
>             cp ${APP}.html ${PREFIX}/share/doc/${APP}.html
>             sk-generate-omf --omf:identifier \
> 	            ${PREFIX}/share/doc/${APP}.html ${APP}.omf
>             sk-install-omf --omf-dir ${SK_OMF_DIR} ${APP}.omf
> 
> All this does is take the existing ${APP}.omf file, and uses
> sk-generate-omf to rewrite the IDENTIFIER in the OMF file to make sure
> it points to wherever the file has been installed.
> 
> This (AFAICS) completely solves the problem of relocatable packages.  We
> just provide a command line interface for editing OMF files.
> 
> For documents that have no OMF (i.e., the author didn't provide one), 
> sk-generate-omf can be used to generate one that SK can use.

I think at first, ScrollKeeper will only support DocBook SGML/XML (which I
think is a decent goal for '1.0' (then from the experience with that we can
possible tackle other (more difficult) file formats)).

If a document does not have an OMF file, we should not attempt to "guess"
anything in any way. This document should not be cataloged/indexed.
ScrollKeeper is here to handle /OMF/ files, if they are not present then
obviously you shouldn't use Scrollkeeper. Even if we try to "guess" the
information, we can't figure out things like Author name, language, category,
version #, document title, etc. (esp. if ScrollKeeper is to handle things like
HTML or plain text).

> > > > Right now we just number them, which guarantees a unique identifier.  
> > > 
> > > Two problems with this:
> > > 
> > >   1.  Probably won't scale efficiently as the Contents List gets larger.
> > > 
> > >   2.  Makes it impossible to merge two separate Contents Lists, from two
> > >       or more sites (say you're on a univesity campus, where the biology
> > >       department and the chemistry department both make available their
> > >       list of documentation in two different Scroll Keeper Contents Lists).
> > 
> > When sk merges seperate databases, it can easily mark them so that it
> > knows where the pieces came.  One way to do this would be to append a few
> > digit code to the beginning of each id which describes which database it
> > came from.
> 
> Yeah, but that's going to needlessly complicate the mechanism for
> merging databases.
> 
> The Contents List is in XML, right.  XML already has a mechanism for
> saying that a file includes other files.
> 
>   <?xml version="1.0">
>   <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
> 
>   <!ENTITY contents.bio  SYSTEM "/path/to/biology/contents/list">
>   <!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">
> 
>   ]>
> 
>   <contents>
>     &contents.bio;
> 
>     &contents.chem;
>   </contents>
> 
> Why reinvent the wheel?  All we need to do is ensure (as reasonably as
> possible) that the identifiers in the different contents lists are
> unique.

Actually - this is a VERY good idea. In fact, DV just suggested this idea on
IRC which I didn't quite understand (then I read this example and I get it)...

Basically, you can have a /etc/scrollkeeper.conf file that lists all the
directories which Scrollkeeper will scan and include as entities.

I have a question though. How will this work for user's directories? Does
Scrollkeeper need to support this "indepenendently (and track it with a
~/.scrollkeeperrc)

> > >       Your document browser is going to need a mechanism to uniquely
> > >       identify each document.  If there's a strong probability that the
> > >       document IDs are not unique then the browsers are each going to
> > >       have to reinvent their own mechanism for ensuring that the
> > >       documents are unique.
> > >
> > >      The documents URL might be appropriate, but it's a bit unwieldy.
> > 
> > The document URL is a very good unique identifier, and it is already being
> > passed to the help browser.
> 
> Yes, it's probably the best thing to use.

URLs are very unreliable and subject to change actually. There are not fixed
forever. You can just move them to another directory and "poof". Servers can
shutdown, etc.

[snipped the rest in an effor to conserve size]

Re: [Scrollkeeper-devel] Category Trees

[Ali A. <ali...@au...>]

* Dan Mueth (d-...@uc...) wrote at 03:43 on 27/11/00:
> 
> On Sat, 25 Nov 2000, Ali Abdin wrote:
> 
> > * Dan Mueth (d-...@uc...) wrote at 01:48 on 25/11/00:
> > > 
> > > On Fri, 24 Nov 2000, Laszlo Kovacs wrote:
> > > 
> > > > How do we deal with category trees? I see category trees as generic
> > > > empty content lists.
> > > > 
> > > > These are the possibilities we discussed:
> > > > 
> > > > 1. Enforce a category tree and ignore all docs that don't fit into it.
> > > > 2. Enforce a tree and put all docs that don't fit into an Other section
> > > > of the Content List.
> > > > 3. Supply a tree (installed) and OMF files come with new categories,
> > > > extend the Categories in the Content List with them (don't modify the
> > > > category tree).
> > > > 4. Recommend a set of categories on the project web site, but don't
> > > > supply any tree. Build the content list(s) from whatever comes in the
> > > > OMF files.
> > > > 5. Don't recommend any categorization, but keep support in the OMF files
> > > > to specify categories. Extend the Content List with whatever comes in
> > > > the OMF files.
> > > > 
> > > > 1. and 5. are there for completeness only. I suppose nobody thinks they
> > > > would be a way to go.
> > > > 
> > > > So what do people think? Which is the best way to do it? Any other
> > > > ideas?
> > > 
> > > One more:
> > > 6) Support more than one of the above and allow the user/SA to decide
> > > which one method is employed in /etc/scrollkeeper.conf or
> > > $HOME/.scrollkeeperrc.
> > > 
> > > One other issue which should be discussed for options #1-#3 - What do we
> > > do with empty parts of the tree?  Do we show branches that don't have
> > > documents in them?  Or are these branches only created as they get
> > > populated?   I think the assumption right now is that empty branches
> > > exist and are passed to the browser.  (The browser could still potentially
> > > snip them out.)
> > 
> > I disagree. I do not think it is worth the complexity to actually let the user
> > choose.
> > 
> > You will also confuse system administrators, and desktop users who are not
> > sure of which setting to you use.
> > 
> > I prefer method 4. If somebody has a reason why we should NOT use number 4,
> > please say why? ;)
> 
> My concerns were:
> 
> 1) We want to have some configurability on the part of the SA or
> user.  This basically means removing branches or adding branches.  For #4,
> adding branches is trivial.  We could potentially provide another
> mechanism for removing branches in place of specifying the tree, so #4
> could still work here.

You don't need to "remove" or "add" branches within scrollkeeper. It will all
be queried at runtime.
 
> 2) Things becoming a mess.  If we use #4, we would still need to provide a
> really good tree on the web page and strongly encourage people to use it
> unless their document really doesn't fit into the existing scheme.  Even
> in this case, they should suggest that the tree be extended in addition to
> creating their own branch in their OMF file.  If we do provide a really
> good tree and most people follow it, then we should be okay.  We will need
> to track down people who have blatently duplicate existing sections in
> other positions than the existing sections.  It only takes a few people
> doing this to produce a very confusing and inconsistant interface for the
> user. I really don't want to see 9 different places for the GIMP in the
> tree ;)

I think that if people "do their own thing" that proves that the Category tree
provided is somehow ineffective (confusing? too many duplicates? etc.) 

I seriously doubt people will just "do their own thing" because they just feel
like it...Actually, on second thoughts, I recommend we DO distribute a
Category list and scrollkeeper-install should provide a "warning" if it is not
ine one of the "suggested" list of categories! (otherwise people will be too
lazy to go to the website to look it up!)

> 3) Typos and variations on translations.  Unless we do #2, we will always
> get spurious duplicated branches from these.  

This is interesting (translations of categories). My vote: don't translate the
category. (this is the same as saying don't translate the XML tags)
 
> Given this, #4 seems like a reasonable solution. The people I've spoken
> with about this so far seem pretty much split between #4 and #2.  #2
> guarantees that concerns (2) and (3) above do not become real problems.  
> Of course it has a couple of its own problems to contribute such as the
> need to keep your contents list categories up-to-date and make the central
> list well-managed and respected (we have to do that anyway though or even
> with #4 we will be screwed.)
> 
> The difference between this and #3 is that #3 will have empty branches.  
> I haven't decided whether I think this is a good thing or a bad thing.  
> If the tree on the web page is huge and has large sections which are
> frequently empty, it may be best not to have them. 
> 
> Does anybody have an opinion on this?

Empty branches suck. They are wasteful. 

Re: [Scrollkeeper-devel] Handling relocatable packages

[Ali A. <ali...@au...>]

* Nik Clayton (ni...@no...) wrote at 03:49 on 26/11/00:
> On Thu, Nov 23, 2000 at 08:05:16PM -0200, Ali Abdin wrote:
> > GNOME is non-relocatable...Nautilus is non-relocatable (I maybe incorrect, 
> > somebody told me they were non-relocatable)
> 
> Gnome might be non-relocatable on a Linux system.  That's not the case
> on a FreeBSD system.  All the world is not Linux.
> 
> [...]

So we should support FreeBSD and not Linux? (this argument works both ways). I
am just saying we should support both relocatable and non-relocatable packages
(you asked for an example of non-reloctabale package, and I gave you one
(GNOME on linux)).
 
> > Now, I know there are probably hacks that can be done to get around this. I
> > know that this may not be "theoretically" correct, etc. etc. etc. But the fact
> > is there /ARE/ non-relocatable packages out there, and scrollkeeper should
> > support both types of packages.
> 
> SK should not need to do anything different, once the data has been
> installed.  We may need to provide tools that make it easy to rewrite an
> existing OMF file if a document is being installed elsewhere (see my
> description of sk-generate-omf in another message).

[snip]
 
> > I am saying just a HTML file, or plain text file just LISTING the categories -
> > nothing fancy...Scrollkeeper would have no knowledge of this file and will not
> > interact with it in anyway.
> 
> XML, probably.  
> 
> SK would need to know about the file, so that it can do sanity checks,
> and say things like
> 
>      The document you are installed lists a category that does not exist
>      in the master category list.  Do you want to continue?
> 
> (or something like that).

Yeah...although I'd still like a HTML and/or plain text version (just to avoid
the visual clutter of the tags). Wait, actually Mozilla now can decently view
XML files so that may not be a problem ;) but whatever.

> > The scrips should NEVER EVER become "interactive". Just use whatever is in the
> > category (in the OMF file).
> 
> People make typos.  SK should probably have a 'loose' and 'strict'
> processing option.  It should allow people to shoot themselves in the
> foot if they want to.

Well - by default, I recommend non-interative prompts..
 
> > > > >   application will have the responsibility for handling user queries of the
> > > > >   SK TOC and Index files ]
> > > > 
> > > > sk should export searching functionality. 
> > > 
> > > How?  libsk.so?  By having a program that accepts search parameters and
> > > returns XML with the results?  Some other mechanism?
> > 
> > I prefer a library that returns the xmlDocPtr of the file...
> 
> xmlDocPtr is specifc to libxml, isn't it?  What if I (as an application
> programmer) want to use another XML library?  Since (part of) the point
> behind XML is to reduce reliance on one specific parsing library, it
> would be more open-ended if libsk could be used to find a document,
> which can then be referred to by URL.  It's then up to the calling
> application to open that URL and process it.

ScrollKeeper /has/ to pick an XML library to rely on for parsing stuff. I was
just suggesting "publicizing" this in the API...Anyway, you /could/ have
libxml-specific stuff trapped in #ifdef's and just provide "other" people with
the FILE descriptor to do their own parsing.

Or, we can make libsk.so libxml-specific totally and then force people who
wish to use something else to use the scripts and/or command-line tools.

Note: to compile (and run) scrollkeeper you will need to have libxml
installed.

Re: [Scrollkeeper-devel] Category Trees

[Laszlo K. <las...@su...>]

> One other issue which should be discussed for options #1-#3 - What do we
> do with empty parts of the tree?  Do we show branches that don't have
> documents in them?  Or are these branches only created as they get
> populated?   I think the assumption right now is that empty branches
> exist and are passed to the browser.  (The browser could still potentially
> snip them out.)

I don't think this is important from architectural point of view at the
moment. This is why we chopped the big email to small emails and now we
are growing it again:-)

Laszlo

Re: [Scrollkeeper-devel] Category Trees

[Dan M. <d-...@uc...>]

On Sat, 25 Nov 2000, Ali Abdin wrote:

> * Dan Mueth (d-...@uc...) wrote at 01:48 on 25/11/00:
> > 
> > On Fri, 24 Nov 2000, Laszlo Kovacs wrote:
> > 
> > > How do we deal with category trees? I see category trees as generic
> > > empty content lists.
> > > 
> > > These are the possibilities we discussed:
> > > 
> > > 1. Enforce a category tree and ignore all docs that don't fit into it.
> > > 2. Enforce a tree and put all docs that don't fit into an Other section
> > > of the Content List.
> > > 3. Supply a tree (installed) and OMF files come with new categories,
> > > extend the Categories in the Content List with them (don't modify the
> > > category tree).
> > > 4. Recommend a set of categories on the project web site, but don't
> > > supply any tree. Build the content list(s) from whatever comes in the
> > > OMF files.
> > > 5. Don't recommend any categorization, but keep support in the OMF files
> > > to specify categories. Extend the Content List with whatever comes in
> > > the OMF files.
> > > 
> > > 1. and 5. are there for completeness only. I suppose nobody thinks they
> > > would be a way to go.
> > > 
> > > So what do people think? Which is the best way to do it? Any other
> > > ideas?
> > 
> > One more:
> > 6) Support more than one of the above and allow the user/SA to decide
> > which one method is employed in /etc/scrollkeeper.conf or
> > $HOME/.scrollkeeperrc.
> > 
> > One other issue which should be discussed for options #1-#3 - What do we
> > do with empty parts of the tree?  Do we show branches that don't have
> > documents in them?  Or are these branches only created as they get
> > populated?   I think the assumption right now is that empty branches
> > exist and are passed to the browser.  (The browser could still potentially
> > snip them out.)
> 
> I disagree. I do not think it is worth the complexity to actually let the user
> choose.
> 
> You will also confuse system administrators, and desktop users who are not
> sure of which setting to you use.
> 
> I prefer method 4. If somebody has a reason why we should NOT use number 4,
> please say why? ;)

My concerns were:

1) We want to have some configurability on the part of the SA or
user.  This basically means removing branches or adding branches.  For #4,
adding branches is trivial.  We could potentially provide another
mechanism for removing branches in place of specifying the tree, so #4
could still work here.

2) Things becoming a mess.  If we use #4, we would still need to provide a
really good tree on the web page and strongly encourage people to use it
unless their document really doesn't fit into the existing scheme.  Even
in this case, they should suggest that the tree be extended in addition to
creating their own branch in their OMF file.  If we do provide a really
good tree and most people follow it, then we should be okay.  We will need
to track down people who have blatently duplicate existing sections in
other positions than the existing sections.  It only takes a few people
doing this to produce a very confusing and inconsistant interface for the
user. I really don't want to see 9 different places for the GIMP in the
tree ;)

3) Typos and variations on translations.  Unless we do #2, we will always
get spurious duplicated branches from these.  

Given this, #4 seems like a reasonable solution. The people I've spoken
with about this so far seem pretty much split between #4 and #2.  #2
guarantees that concerns (2) and (3) above do not become real problems.  
Of course it has a couple of its own problems to contribute such as the
need to keep your contents list categories up-to-date and make the central
list well-managed and respected (we have to do that anyway though or even
with #4 we will be screwed.)

The difference between this and #3 is that #3 will have empty branches.  
I haven't decided whether I think this is a good thing or a bad thing.  
If the tree on the web page is huge and has large sections which are
frequently empty, it may be best not to have them. 

Does anybody have an opinion on this?

Dan

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Thu, Nov 23, 2000 at 08:05:16PM -0200, Ali Abdin wrote:
> GNOME is non-relocatable...Nautilus is non-relocatable (I maybe incorrect, 
> somebody told me they were non-relocatable)

Gnome might be non-relocatable on a Linux system.  That's not the case
on a FreeBSD system.  All the world is not Linux.

[...]

> Now, I know there are probably hacks that can be done to get around this. I
> know that this may not be "theoretically" correct, etc. etc. etc. But the fact
> is there /ARE/ non-relocatable packages out there, and scrollkeeper should
> support both types of packages.

SK should not need to do anything different, once the data has been
installed.  We may need to provide tools that make it easy to rewrite an
existing OMF file if a document is being installed elsewhere (see my
description of sk-generate-omf in another message).

<snip>

> So, multiple databases does make sense in my opinion (think of it this way,
> just because YOU don't need it, or can't find a reason to use it, it doesn't
> mean that it is not useful for others)..

Exactly.  Tools, not policy.

> FAQ/Manual/HOWTO are not category types in your opinion...What I mean by
> categories is more like "subject" - i.e. "Audio/MP3 Players' or
> 'Office/Spreadsheet' or 'X11/Internet/FTP Clients' etc.

Maybe, maybe not.  SK shouldn't care.  Categories should be opaque types
to it.  SK should place no special meaning on any category.

> I am saying just a HTML file, or plain text file just LISTING the categories -
> nothing fancy...Scrollkeeper would have no knowledge of this file and will not
> interact with it in anyway.

XML, probably.  

SK would need to know about the file, so that it can do sanity checks,
and say things like

     The document you are installed lists a category that does not exist
     in the master category list.  Do you want to continue?

(or something like that).

> The scrips should NEVER EVER become "interactive". Just use whatever is in the
> category (in the OMF file).

People make typos.  SK should probably have a 'loose' and 'strict'
processing option.  It should allow people to shoot themselves in the
foot if they want to.

> > > >   application will have the responsibility for handling user queries of the
> > > >   SK TOC and Index files ]
> > > 
> > > sk should export searching functionality. 
> > 
> > How?  libsk.so?  By having a program that accepts search parameters and
> > returns XML with the results?  Some other mechanism?
> 
> I prefer a library that returns the xmlDocPtr of the file...

xmlDocPtr is specifc to libxml, isn't it?  What if I (as an application
programmer) want to use another XML library?  Since (part of) the point
behind XML is to reduce reliance on one specific parsing library, it
would be more open-ended if libsk could be used to find a document,
which can then be referred to by URL.  It's then up to the calling
application to open that URL and process it.

> > For example, on a BSD system, the contents of /usr/share/man change once
> > in a blue moon, as third party manual pages generally live under
> > /usr/local/share/man.  So the admin might want to create a Contents List
> > for /usr/share/man once, and keep it separate, because the SA knows its
> > never going to change, so it doesn't need to be included in any
> > processes SK undertakes to ensure that the Contents List is sync with
> > the filesystem.
> 
> I think this "optimization" is unncessary at this time. We have no data _AT
> ALL_ on the efficiency in scrollkeeper. We don't know if it will be slow or
> fast, so splitting up databases for this reason is unacceptable in my opinion.

I wasn't introducing it is an example of an optmisation, but as an
example of where an SA might want to have separate databases.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Wed, Nov 22, 2000 at 09:55:45PM -0600, Dan Mueth wrote:
> > Can you give me an example of non-relocatable package, and it's
> > justification for being non-relocatable?
> 
> On Red Hat `rpm -qai | grep "Relocations: (not relocateable)" | wc`
> gives me 506 packages which are non-relocateable, the last of which is
> xbill:)
> 
> I think Red Hat would be better qualified at explaining this decision.

Someone else bought up Gnome as an example of non-relocatable package.

Just because the original implementors didn't provide a hook to make it
non-relocatable, don't assume that other's won't have done.

> I am still uncertain if we want inheritance between contents lists though.
> 
> In many cases users will not want inheritance, because they lose any
> control over what they inherit.  

"many cases" means, therefore, that there will be times when users will
want inheritence.

There's a saying that's prevalent in the BSD world; "mkae tools, not policy".
Having read some of the later messages in this thread, I'm going to be
coming back to that quite a lot.

I don't care what I, personally, might need some sk.  However, it's
important not to impose artificial limitations that impose my (or your)
policy for how to configure a host/network on someone else.

> We definitely want libsk.so.  I'm not sure if we would want a command-line
> program for this.

It's not something we need (if someone else needs it then they can write 
it).  It'll probably prove to be a useful small sample application
though.

<snip>

> Aside from rebuilding the database in cases where it becomes corrupt, I'd
> like this to be a zero maintenance system that works out of the box.  I
> don't think we should focus on unusual things a SA may want to
> do.  

"Tools, not policy" (see, I said I'd be coming back to it).  The things
that you are I think are unusual might be standard practice for someone
who wants to use sk.

> > So if you have multiple Contents Lists, you need to have multiple ToCs
> > (and Indexes), and a way of saying that >this< Contents List was used to
> > produce >This< ToC.
> 
> Not if only one contents list/toc pair live in each database directory.

Tools, not policy.

I might want to put the Contents List in one directory (which is
private, and not available to the world), and publish a ToC.

We can certainly say

   If you don't tell it otherwise, sk will generate the ToC in the same
   place as the Contents List.

but we should make it possible for the SA to change this behaviour if
necessary.

> > Do you agree that making the distinction between a document, and one or
> > more instances of that document (different formats, different languages)
> > is worthwhile.
> 
> The only way scrollkeeper knows that two docs of different
> versions/formats/languages are otherwise the same is using the extra OMF
> element (which doesn't yet exist) which assigns them an ISBN-type code
> which is the same.  Since they have different IDENTIFIERS (and other
> metadata), sk knows they are different documents.

This is (possibly) something that can be handled outside of sk with
creative use of the RELATION element in the OMF.

> > > Why do we specify the format on the command line?  The FORMAT is specified
> > > in the OMF already.  The packager should just install and register the
> > > metadata instead of putting it on the command line.
> > 
> > Because at this point, all we've done is install a document with no OMF.
> > SK has to synthesise some OMF, based on any additional information we 
> > give it.
> > 
> > This separation is necessary to support documents that have no separate
> > OMF.  Adding "-format" to the command line is probably going to be
> > easier for an end user than having to create a separate file to return
> > to.
> 
> Perhaps we agree on this after all and are just talking past each
> other.  So for docs with OMF we have two steps:
> 1) update the IDENTIFIER to point to the right URL
> 2) register the OMF file with sk
> 
> Really, #1 is only necessary for local docs (as opposed to remote
> ones) and for relocatable packages.

I think that's right.  In my example, "sk-install-doc" is probably the
wrong name.

Ah ha, I think I've got a work around.

We've been going back and forth about what to do if a document doesn't
have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
wrong, or whatever.

So, what if we have two command, sk-generate-omf and sk-install-omf?  The 
usage for this would look something like;

  sk-generate-omf [--omf:creator ...]
	  	  [--omf:maintainer ...]
		  [--omf:contributor ...]
		  [--omf:title ...]
		  [--omf:date ...]
                  [<other --omf:... options]
		  file.omf

  sk-install-omf [--omf-dir ...] /path/to/omf/file

sk-generate-omf is used to generate OMF for installation into sk.  It can be 
told about an existing OMF file (for example, if the document has been 
distributed with an OMF file).  However, it also has a series of --omf:<foo>
options, which are used to override the OMF in the .omf file, and replace 
it with something else (I haven't listed them all in the synopsis above).

sk-install-omf is used to install an OMF file into SK.

In the simplest case (where the document already has a separate OMF
file), in the installation Makefile, this might look something like

    PREFIX=		/usr/local
    APP=		appname
    SK_OMF_DIR=		`sk-config --omf-dir`

    ...

    install-doc:
            cp ${APP}.html ${PREFIX}/share/doc/${APP}.html
            sk-generate-omf --omf:identifier \
	            ${PREFIX}/share/doc/${APP}.html ${APP}.omf
            sk-install-omf --omf-dir ${SK_OMF_DIR} ${APP}.omf

All this does is take the existing ${APP}.omf file, and uses
sk-generate-omf to rewrite the IDENTIFIER in the OMF file to make sure
it points to wherever the file has been installed.

This (AFAICS) completely solves the problem of relocatable packages.  We
just provide a command line interface for editing OMF files.

For documents that have no OMF (i.e., the author didn't provide one), 
sk-generate-omf can be used to generate one that SK can use.

> > > Right now we just number them, which guarantees a unique identifier.  
> > 
> > Two problems with this:
> > 
> >   1.  Probably won't scale efficiently as the Contents List gets larger.
> > 
> >   2.  Makes it impossible to merge two separate Contents Lists, from two
> >       or more sites (say you're on a univesity campus, where the biology
> >       department and the chemistry department both make available their
> >       list of documentation in two different Scroll Keeper Contents Lists).
> 
> When sk merges seperate databases, it can easily mark them so that it
> knows where the pieces came.  One way to do this would be to append a few
> digit code to the beginning of each id which describes which database it
> came from.

Yeah, but that's going to needlessly complicate the mechanism for
merging databases.

The Contents List is in XML, right.  XML already has a mechanism for
saying that a file includes other files.

  <?xml version="1.0">
  <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [

  <!ENTITY contents.bio  SYSTEM "/path/to/biology/contents/list">
  <!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">

  ]>

  <contents>
    &contents.bio;

    &contents.chem;
  </contents>

Why reinvent the wheel?  All we need to do is ensure (as reasonably as
possible) that the identifiers in the different contents lists are
unique.

> >       Your document browser is going to need a mechanism to uniquely
> >       identify each document.  If there's a strong probability that the
> >       document IDs are not unique then the browsers are each going to
> >       have to reinvent their own mechanism for ensuring that the
> >       documents are unique.
> >
> >      The documents URL might be appropriate, but it's a bit unwieldy.
> 
> The document URL is a very good unique identifier, and it is already being
> passed to the help browser.

Yes, it's probably the best thing to use.

> > You really need a mechanism that can (as near as possible) guarantee
> > globally unique document IDs.
> 
> I'm pretty sure that only one document lives at a given URL ;)

Not necessarily.  Web servers have been known to serve up different
document depends on the settings of the browser requesting the document
(for example, it's list of acceptable languages).

> > True.  However, we should allow the Index builder flexibilit in deciding
> > what documents are included in the Index.  This may include producing a
> > special Index that only contains documents available in HTML, plain
> > text, and PDF (for example).
> 
> What do you mean by "index"?  The definition I gave for "index" in prop#3
> is what you find in the back of a book - an alphabetical list of concepts
> and links to where they are discussed in the given work.  My impression is
> that you mean the Contents List here.  Is that right?

Yes, my mistake.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Category Trees

[Ali A. <ALI...@au...>]

> At 1:58 AM -0200 11/25/00, Ali Abdin wrote:
> >If we want truly unique id's then we just use the 'uuid-gen'
> >utility found on most machines.
> 
> Its not on FreeBSD (or even mentioned in /usr/ports/INDEX),
> though I'm sure it could be added.

I thought it was part of the 'e2fsprogs' package (well, thats what its 
part of on my RPM-based system). Oh wait, FreeBSD doesn't use ext2 does 
it?
 
> My opinion, however, is that there is no reason to use ugly, long,
> and easily forgotten keys, in pursuit of mechanical perfection,
> when better ones are available.
> 
> More to the point, it is very handy for folks discussing a given
> package to be able to refer to it clearly and unambiguously.  I
> think my proposal serves here, but that the uuid-gen keys do not.

The uuid-gen 'key' is not supposed to be human readable, or referred to 
by humans, it is an "internal" cataloging purpose thing (like ISBN 
number (you don't memorize those, or refer to a book by them, do you?))

If you want to refer to a doc, you would just use the name, version 
number, author, and possibly locale....

The "unique" id will become more important if/when OMF gets wide adoption 
and you have online servers "talking" to each other. (this "key" would be 
most useful if you have an online central repository that you could talk 
to and query)

Regards,
Ali

Re: [Scrollkeeper-devel] Category Trees

[Rich M. <rd...@cf...>]

At 1:58 AM -0200 11/25/00, Ali Abdin wrote:
>If we want truly unique id's then we just use the 'uuid-gen'
>utility found on most machines.

Its not on FreeBSD (or even mentioned in /usr/ports/INDEX),
though I'm sure it could be added.

My opinion, however, is that there is no reason to use ugly, long,
and easily forgotten keys, in pursuit of mechanical perfection,
when better ones are available.

More to the point, it is very handy for folks discussing a given
package to be able to refer to it clearly and unambiguously.  I
think my proposal serves here, but that the uuid-gen keys do not.

-t
-- 
--
Rich Morin:          rd...@cf..., +1 650-873-7841, http://www.cfcl.com/rdm
Prime Time Freeware: in...@pt..., +1 408-433-9662, http://www.ptf.com
MacPerl: http://www.macperl.com, http://www.ptf.com/ptf/products/MPPE
MkLinux: http://www.mklinux.org, http://www.ptf.com/ptf/products/MKLP

Re: [Scrollkeeper-devel] Category Trees

[Ali A. <ali...@au...>]

* Rich Morin (rd...@cf...) wrote at 01:50 on 25/11/00:
> The "Tree of Knowledge Fallacy" has to do with the mistaken belief
> that you can put all of knowledge into a single hierarchy.  Anyone
> who has ever tried to sort miscellaneous items has run into it:
> Lessee now; I seem to have a lot of screws here.  Do I put all of
> the Phillips-head screws together, sort by length, or what?
> 
> The problem clearly manifests itself in the area of Open Source
> software packages.  With FileWatcher tracking some 10K packages,
> it is totally unreasonable to believe that any categorization
> scheme will fit all packages for all needs.  Politics aside
> (my tree's better than your tree...), different sorting schemes
> will always be better or poorer for different searching needs.

Heh, I think you're right. I think Help files SHOULD be able to have multiple
categories! This really does make sense (although potentially confusing for
the user). 

Should the OMF files though have the ability to have "endless" amounts of
categories for a file, or should we impose a limit (i.e. can specify a max of
3 categories, in order from "most appropriate" to "less appropriate" (e.g. for
Nautilus we would have 1) Desktop/File Manager 2) Internet/Web Browser 3)
Audio/MP3 Player (it does play MP3s!!))
 
> So, I suggest that we concentrate on things we _can_ control:
> 
>    *  unique identifiers
> 
>         Each package (and each variant and version of same)
>         should have a unique identifier, so that we can be
>         clear and unambiguous in referring to it.  This is
>         really the same issue as the database folks have...

Yeah, I agree - I disagree with your other mail specifying how to do it. If we
want truly unique id's then we just use the 'uuid-gen' utility found on most
machines. 

>    *  characteristics
> 
>         A package can have any number of characteristics (e.g.,
>         keywords) that someone might use to look it up.  One
>         form of keyword is the package's position in a given
>         hierarchy (e.g., freebsd-ports/devel/pmake).

I'm not sure about this. I don't think we should just add in any features that
might possibly be useful somehow, so we may as well throw it in now. The OMF
file should already have all the metadata necessary for indexing documents - I
don't see why we need a generic "characteristics" or "keywords" section

> By explicitly allowing for the presence of multiple category
> trees, this method is able to incorporate (or even dynamically
> add) any number of categorization methods at the same time.

I don't think we should have a strict "category" tree. We can just allow
people to give each doc multiple categories, but ScrollKeeper should not
enforce it!

Regards,
Ali

Re: [Scrollkeeper-devel] Category Trees

[Ali A. <ali...@au...>]

* Dan Mueth (d-...@uc...) wrote at 01:48 on 25/11/00:
> 
> On Fri, 24 Nov 2000, Laszlo Kovacs wrote:
> 
> > How do we deal with category trees? I see category trees as generic
> > empty content lists.
> > 
> > These are the possibilities we discussed:
> > 
> > 1. Enforce a category tree and ignore all docs that don't fit into it.
> > 2. Enforce a tree and put all docs that don't fit into an Other section
> > of the Content List.
> > 3. Supply a tree (installed) and OMF files come with new categories,
> > extend the Categories in the Content List with them (don't modify the
> > category tree).
> > 4. Recommend a set of categories on the project web site, but don't
> > supply any tree. Build the content list(s) from whatever comes in the
> > OMF files.
> > 5. Don't recommend any categorization, but keep support in the OMF files
> > to specify categories. Extend the Content List with whatever comes in
> > the OMF files.
> > 
> > 1. and 5. are there for completeness only. I suppose nobody thinks they
> > would be a way to go.
> > 
> > So what do people think? Which is the best way to do it? Any other
> > ideas?
> 
> One more:
> 6) Support more than one of the above and allow the user/SA to decide
> which one method is employed in /etc/scrollkeeper.conf or
> $HOME/.scrollkeeperrc.
> 
> One other issue which should be discussed for options #1-#3 - What do we
> do with empty parts of the tree?  Do we show branches that don't have
> documents in them?  Or are these branches only created as they get
> populated?   I think the assumption right now is that empty branches
> exist and are passed to the browser.  (The browser could still potentially
> snip them out.)

I disagree. I do not think it is worth the complexity to actually let the user
choose.

You will also confuse system administrators, and desktop users who are not
sure of which setting to you use.

I prefer method 4. If somebody has a reason why we should NOT use number 4,
please say why? ;)
 
> And another issue: If we have a categories tree, should we allow the user
> to specify in the conf file which tree to use?  Presumably one may want to
> use an alternate tree, such as a subset of the recommended tree.  For
> example, a user might copy the default tree into
> $HOME/.scrollkeeper/categories.xml and then chop out the man and info
> page sections.

Well, I vote no category tree :P

Re: [Scrollkeeper-devel] Category Trees

[Rich M. <rd...@cf...>]

I didn't say much about the exact nature of the "unique identifiers",
largely because I don't have anything _that_ wonderful to propose.  I
have done some thinking on the subject, however, and I think that the
identifiers should be as mnemonic as possible, consistent with the
need for precision.

So, I might start with the "common name" of the package, coupled
with the initials of the principal developer, as:

   icon-rg
   perl-lw
   sendmail-ea

If we find a "sendmail" package written by Erin Aardvark, we can
call it "sendmail-ea2", but I don't see this happening too often.
It is a fairly simple experiment to get the names of the packages
listed by FileWatcher and sort them without respect to case.  YOU
can then examine the result for clashes.  I have, and there really
aren't all that many (after you allow for variants and versions of
the same package).

-r
-- 
--
Rich Morin:          rd...@cf..., +1 650-873-7841, http://www.cfcl.com/rdm
Prime Time Freeware: in...@pt..., +1 408-433-9662, http://www.ptf.com
MacPerl: http://www.macperl.com, http://www.ptf.com/ptf/products/MPPE
MkLinux: http://www.mklinux.org, http://www.ptf.com/ptf/products/MKLP

Re: [Scrollkeeper-devel] Category Trees

[Rich M. <rd...@cf...>]

The "Tree of Knowledge Fallacy" has to do with the mistaken belief
that you can put all of knowledge into a single hierarchy.  Anyone
who has ever tried to sort miscellaneous items has run into it:
Lessee now; I seem to have a lot of screws here.  Do I put all of
the Phillips-head screws together, sort by length, or what?

The problem clearly manifests itself in the area of Open Source
software packages.  With FileWatcher tracking some 10K packages,
it is totally unreasonable to believe that any categorization
scheme will fit all packages for all needs.  Politics aside
(my tree's better than your tree...), different sorting schemes
will always be better or poorer for different searching needs.

So, I suggest that we concentrate on things we _can_ control:

   *  unique identifiers

        Each package (and each variant and version of same)
        should have a unique identifier, so that we can be
        clear and unambiguous in referring to it.  This is
        really the same issue as the database folks have...

   *  characteristics

        A package can have any number of characteristics (e.g.,
        keywords) that someone might use to look it up.  One
        form of keyword is the package's position in a given
        hierarchy (e.g., freebsd-ports/devel/pmake).

By explicitly allowing for the presence of multiple category
trees, this method is able to incorporate (or even dynamically
add) any number of categorization methods at the same time.

-r

-- 
--
Rich Morin:          rd...@cf..., +1 650-873-7841, http://www.cfcl.com/rdm
Prime Time Freeware: in...@pt..., +1 408-433-9662, http://www.ptf.com
MacPerl: http://www.macperl.com, http://www.ptf.com/ptf/products/MPPE
MkLinux: http://www.mklinux.org, http://www.ptf.com/ptf/products/MKLP

Re: [Scrollkeeper-devel] Category Trees

[Dan M. <d-...@uc...>]

On Fri, 24 Nov 2000, Laszlo Kovacs wrote:

> How do we deal with category trees? I see category trees as generic
> empty content lists.
> 
> These are the possibilities we discussed:
> 
> 1. Enforce a category tree and ignore all docs that don't fit into it.
> 2. Enforce a tree and put all docs that don't fit into an Other section
> of the Content List.
> 3. Supply a tree (installed) and OMF files come with new categories,
> extend the Categories in the Content List with them (don't modify the
> category tree).
> 4. Recommend a set of categories on the project web site, but don't
> supply any tree. Build the content list(s) from whatever comes in the
> OMF files.
> 5. Don't recommend any categorization, but keep support in the OMF files
> to specify categories. Extend the Content List with whatever comes in
> the OMF files.
> 
> 1. and 5. are there for completeness only. I suppose nobody thinks they
> would be a way to go.
> 
> So what do people think? Which is the best way to do it? Any other
> ideas?

One more:
6) Support more than one of the above and allow the user/SA to decide
which one method is employed in /etc/scrollkeeper.conf or
$HOME/.scrollkeeperrc.

One other issue which should be discussed for options #1-#3 - What do we
do with empty parts of the tree?  Do we show branches that don't have
documents in them?  Or are these branches only created as they get
populated?   I think the assumption right now is that empty branches
exist and are passed to the browser.  (The browser could still potentially
snip them out.)

And another issue: If we have a categories tree, should we allow the user
to specify in the conf file which tree to use?  Presumably one may want to
use an alternate tree, such as a subset of the recommended tree.  For
example, a user might copy the default tree into
$HOME/.scrollkeeper/categories.xml and then chop out the man and info
page sections.

Dan

[Scrollkeeper-devel] Category Trees

[Laszlo K. <las...@su...>]

Hi,

here is another issue from our very long email. 

How do we deal with category trees? I see category trees as generic
empty content lists.

These are the possibilities we discussed:

1. Enforce a category tree and ignore all docs that don't fit into it.
2. Enforce a tree and put all docs that don't fit into an Other section
of the Content List.
3. Supply a tree (installed) and OMF files come with new categories,
extend the Categories in the Content List with them (don't modify the
category tree).
4. Recommend a set of categories on the project web site, but don't
supply any tree. Build the content list(s) from whatever comes in the
OMF files.
5. Don't recommend any categorization, but keep support in the OMF files
to specify categories. Extend the Content List with whatever comes in
the OMF files.

1. and 5. are there for completeness only. I suppose nobody thinks they
would be a way to go.

So what do people think? Which is the best way to do it? Any other
ideas?

Laszlo

[Scrollkeeper-devel] Multiple Databases

[Laszlo K. <las...@su...>]

Hi All,

I will try to rip the big email to several smaller emails. Here is one,
the issue of multiple databases. Let's ignore whatever we have designed
and implemented for now and try to gather together why do we think this
would be good.

Some ideas I write up here come from the architecture of Solaris/CDE
Help. Some of the ideas below are very different from the current
design:-)

The idea of accessing various databases across the internet looked very
interesting for everybody. I think this is a good idea.

Whereever a remote database is, there has to be a Scrollkeeper around. I
would expect that remote database to be up-to-date, I don't think my
local Scrollkeeper should be responsible for this, it should take for
granted whatever is offered by the remote database. In this case
database would mean extracted TOC, Index, etc I would expect all of this
to come through the remote connection. How does that sound?

This will obviously need a script/binary/config file that will tell
Scrollkeeper that which remote databases should be considered.

The model would be the same for locally installed docs. More databases
on the local system and a script/binary/config file tell which ones to
take into account.

Now this is where it becomes rough, I suggest to change the current
design a lot:-)

1. We install the OMF files per package in the package directory (like
/usr/share/pkgname/omf).
2. We say Scrollkeeper is needed to build the package.
3. We generate partial content list file for this package at build time
(obviously it has to be very different from the one we have now). Same
for TOC, Index etc.
4. At install time we create symbolic link of this package to a special
directory (/etc/scrollkeeper or similar). These directories can be
specified through a search path (more than one dir).
5. We provide an interface that merges the partial lists into a main
list and stores it in $/home/.scrollkeeper. We use timestamps to monitor
the partial database changes (only the files have to be checked).

This is it. This would have some advantages, but many disadvantages
also. So please go ahead and point out the problems.

As about merging, I don't see that a big problem.

Laszlo

Re: [Scrollkeeper-devel] Handling relocatable packages

[Dan M. <d-...@uc...>]

On Thu, 23 Nov 2000, Ali Abdin wrote:

> Too be honest, I got tired of reading this very very very long mail (it's been
> one hour!!!!)
> 
> Anyway - I think somebody needs to come up with SPECIFIC suggestions, and
> split this really long thread/mail into separate smaller, more manageable
> threads.

Thanks Ali, Nik, and everybody else who has been reading and/or replying
to these very long emails.  I think it is a great idea to distill the
biggest unresolved issues into seperate shorter threads.

Dan