Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@Su...>]

> Here are a few of my thoughts on what sort of functionality a user may
> want.
> 
> 1) "Basic Document Index"
> 
> For any document, a user may want an index.  Here I mean "index" as what
> one normally finds at the back of a book.  It is a list of terms, often
> with secondary or even tertiary headings and "see also" items.  These
> terms link to concepts and/or words in the body of the document.  The
> purpose is to allow a user who is looking for information on a particular
> concept to quickly find the part(s) of the document which discuss this
> concept.
> 
> DocBook provides tags to markup terms to appear in an index.  When the
> document is processed, such as when one converts the SGML to HTML or PS,
> it generates the index from the many marked up terms throughout the
> document.  For HTML, a link is provided from the index to the correct part
> of the document.  For PS, I believe the page number is listed.
> 
> This is a very useful level of indexing which is already possible without
> ScrollKeeper.  The only requirements are that an experienced indexer
> manually add the indexing markup and that the document processor knows how
> to generate the index.  (Note this requires a fair amount of expertise and
> time on the part of the indexer, but I believe the technology is complete
> for DocBook/SGML documents rendered into the standard target formats.)
> 
> 2) "Help System Aware Document Index"
> 
> To increase usability, the help browser is aware of the document index
> from #1, rather than merely displaying the content of the index.  This
> allows the user to do things like:
> 
> a) browse the index in one window or portion of the help browser while
> viewing the document in another
> 
> b) more quickly navigate and search the index using an index browser built
> into the help system
> 
> c) search multiple documents to identify which one discusses a certain
> topic, or even create compound indexes.
> 
> d) (many other things which are possible, but which probably should be
> done in ScrollKeeper instead of the help browser.  I discuss some of them
> below.)
> 
> I am mainly interested in (a) and (b) above, since I think (c) is probably
> something which should be in (d).
> 
> 3) "System-wide Index"
> 
> This is an index of all documents on the system, obtained my merging all
> of the indexes to all of the documents (of a given locale and excluding
> duplicates of a given document) on the system.
> 
> A system-wide index may be valuable for searching for more obscure things,
> like "beowulf".  However, putting in "cd-rom" or "printer" will likely
> return so many items it may be completely useless to the user.
> 
> 4) "Group Indexes"
> 
> This is an index of a certain set of documents which may not span the
> entire set of documents.  For example, it may be an index for all GNOME
> documents, or all KDE documents, or all documents written by John.  I
> believe this will be the most valuable index type.
> 
> In fact, for GNOME I see the need for at least two different groups.  One
> for the core desktop, and another for all GNOME documents.  Similar to the
> example above, it would be nice to enter "printer" and get only a few hits
> about configuring and using printers, not the "how to print from
> application foo" for every application (or every GNOME application) on
> your computer.  So generally I would expect a help browser to request
> indexes from ScrollKeeper for multiple groups of documents.

Sounds good.

> 
> A few thoughts on technical requirements:
> 
> 1) Index Targets
> 
> We need to be able to parse the DocBook/SGML documents and extract
> their indexing terms and anchors.  Identifying the indexing terms should
> be straightforward.  The other half is predicting the anchors in the
> document.  This is another example of where we require our document
> processors to behave in a uniform way.  For example, if we have two
> DocBook to HTML converters, we need them both to label their anchors the
> same way.
> 
> Also, I think we may soon have a good way to render DocBook/XML documents
> directly.  I'm not sure how we would specify the targets in this case.
> (Could anybody provide some information on how this would work?)
> 
> We will need to resolve this issue to achieve any of the 4 functional
> points listed above except #1.

I guess the way how the browser is able or not to jump to the index ID
provided is up to the browser, not Scrollkeeper. We just have to be sure
that we push the usage of those index terms from SGML that facilitate
the implementation of the jump to the right index content.

> 
> 2) General Database and API
> 
> We should make the index database as general as possible.  Ideally, for
> each occurance of each indexing term, we associate all of the OMF metadata
> of the document.  This allows the help browser to make arbitrary queries,
> from requesting an index for a single document, to an index of the whole
> system, to an index of an arbitrary list of documents, to an index with
> certain metadata (eg. documents with a <subject> of "Core GNOME
> Documents").
> 
> This would mean we want to select a free database which ScrollKeeper will
> use to manage all of the indexing data.  The exported ScrollKeeper API for
> returning various indexes would probably be a rather thin wrapper around
> the database.  It also means that ScrollKeeper will leave the innocent
> world of scripts and command-line programs which point to XML files and
> at least the indexing portion will need to be a proper library which
> browsers link to.
> 
> This approach is probably a fair amount more work than using XML files as
> we have in the past.  It will also require more up-front investment in
> terms of design and expertise.  However, I think we need the speed and
> flexibility of a real database in order to provide some of the most
> valuable functionality: The ability to produce indexes on specific groups
> of documents, specified by their metadata.

I think the database is not feasible in the near future. I think the two
most obvious queries are searching for a term in one doc's index and
searching in a group of docs' indexes where the docs are grouped
according to their metadata subject field (the same that is used to add
them to the right place in the category tree). I am sure people can come
up with loads of useful queries, but I think these will be far the most
common. As we have the docs in the category tree ordered by subject,
grouping according to subject is easy. Merging a number of indexes into
a temporary one will not be very difficult. And this is all we need.

The problem with the free database approach is that we dont know which
one is available on the most OS-es, we certainly know that none right
now on Solaris. While this might be sorted in the future, it is
certainly not very close. The approach I described above can be
implemented XML based in 2-3 months and it will work with a not very
large amount of docs. A database approach would take much longer. I dont
exclude the possibility of a parallel XML/database approach or moving
totally to a database though.

Laszlo