Re: [Scrollkeeper-devel] Index support

[Dan M. <da...@ea...>]

Before we get too far into technical details of how to implement indexing,
I'd like to discuss the general ideas for what we want to achieve.  Here I
think we should start from the perspective of the end-user sitting in
front of a help browser.  The help browser I am thinking of is the GNOME
help browser or perhaps the KDE help browser.  However, what we design
must be able to work just as well for a general Linux help browser, *nix
help browser, or web-based help browser for project Foo or organization
Bar.

Here are a few of my thoughts on what sort of functionality a user may
want.

1) "Basic Document Index" 

For any document, a user may want an index.  Here I mean "index" as what
one normally finds at the back of a book.  It is a list of terms, often
with secondary or even tertiary headings and "see also" items.  These
terms link to concepts and/or words in the body of the document.  The
purpose is to allow a user who is looking for information on a particular
concept to quickly find the part(s) of the document which discuss this
concept.

DocBook provides tags to markup terms to appear in an index.  When the
document is processed, such as when one converts the SGML to HTML or PS,
it generates the index from the many marked up terms throughout the
document.  For HTML, a link is provided from the index to the correct part
of the document.  For PS, I believe the page number is listed.

This is a very useful level of indexing which is already possible without
ScrollKeeper.  The only requirements are that an experienced indexer
manually add the indexing markup and that the document processor knows how
to generate the index.  (Note this requires a fair amount of expertise and
time on the part of the indexer, but I believe the technology is complete
for DocBook/SGML documents rendered into the standard target formats.)

2) "Help System Aware Document Index"

To increase usability, the help browser is aware of the document index
from #1, rather than merely displaying the content of the index.  This
allows the user to do things like:

a) browse the index in one window or portion of the help browser while
viewing the document in another

b) more quickly navigate and search the index using an index browser built
into the help system

c) search multiple documents to identify which one discusses a certain
topic, or even create compound indexes.

d) (many other things which are possible, but which probably should be
done in ScrollKeeper instead of the help browser.  I discuss some of them 
below.)

I am mainly interested in (a) and (b) above, since I think (c) is probably
something which should be in (d).

3) "System-wide Index"

This is an index of all documents on the system, obtained my merging all
of the indexes to all of the documents (of a given locale and excluding
duplicates of a given document) on the system.

A system-wide index may be valuable for searching for more obscure things,
like "beowulf".  However, putting in "cd-rom" or "printer" will likely
return so many items it may be completely useless to the user.

4) "Group Indexes"

This is an index of a certain set of documents which may not span the
entire set of documents.  For example, it may be an index for all GNOME
documents, or all KDE documents, or all documents written by John.  I
believe this will be the most valuable index type.

In fact, for GNOME I see the need for at least two different groups.  One
for the core desktop, and another for all GNOME documents.  Similar to the
example above, it would be nice to enter "printer" and get only a few hits
about configuring and using printers, not the "how to print from
application foo" for every application (or every GNOME application) on
your computer.  So generally I would expect a help browser to request
indexes from ScrollKeeper for multiple groups of documents.

--

A few thoughts on technical requirements:

1) Index Targets

We need to be able to parse the DocBook/SGML documents and extract
their indexing terms and anchors.  Identifying the indexing terms should
be straightforward.  The other half is predicting the anchors in the
document.  This is another example of where we require our document
processors to behave in a uniform way.  For example, if we have two
DocBook to HTML converters, we need them both to label their anchors the
same way.  

Also, I think we may soon have a good way to render DocBook/XML documents
directly.  I'm not sure how we would specify the targets in this case.  
(Could anybody provide some information on how this would work?)

We will need to resolve this issue to achieve any of the 4 functional
points listed above except #1.

2) General Database and API

We should make the index database as general as possible.  Ideally, for
each occurance of each indexing term, we associate all of the OMF metadata
of the document.  This allows the help browser to make arbitrary queries,
from requesting an index for a single document, to an index of the whole
system, to an index of an arbitrary list of documents, to an index with
certain metadata (eg. documents with a <subject> of "Core GNOME
Documents").

This would mean we want to select a free database which ScrollKeeper will
use to manage all of the indexing data.  The exported ScrollKeeper API for
returning various indexes would probably be a rather thin wrapper around
the database.  It also means that ScrollKeeper will leave the innocent
world of scripts and command-line programs which point to XML files and
at least the indexing portion will need to be a proper library which
browsers link to.

This approach is probably a fair amount more work than using XML files as
we have in the past.  It will also require more up-front investment in
terms of design and expertise.  However, I think we need the speed and
flexibility of a real database in order to provide some of the most
valuable functionality: The ability to produce indexes on specific groups
of documents, specified by their metadata.

Dan