scrollkeeper-devel

[Scrollkeeper-devel] Index support

[Laszlo K. <las...@Su...>]

Hi,

We've been talking for a while about adding index support to
Scrollkeeper. It looks like we have the people to work on it now, Mary
Dwyer (from Sun), John Fleck and Sasha most of the time. mary would
probably do the development work.

I frame up my idea about how should it be done and we can discuss it.

I dont know much about index tags in DocBook/SGML so I am not sure how
we implement this. I think there are two ways of indexing, one that just
defines the index term and an "anchor" to where it is discussed (or
maybe more anchors?) and the other holds the index definition itself.

Anyway I think the way it should be done is that we take the SGML file
and extract the index info from it into a very simple xml file. Ideally
this should contain only the index strings and the "anchor" in the text
that is associated with them. I dont know if the SGML index tags support
this aproach.

On the technical side I think we should use a SAX interface rather then
a DOM tree as very little info from the SGML docs is actually used. 

The index extractor should be a new binary or part of the Scrollkeeper
library if we manage to turn SK to a library.

This is a very short description, but I think it contains the main
issues.

What do you think about it?

Laszlo

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@Su...>]

I obviously forgot some of the stuff I wanted to mention. If we want to
implement Search later on as part of Scrollkeeper then the index support
should be done keeping in mind that it should be easily searchable later
on. If we leave the search to the browser then we dont need this.

Laszlo

> 
> Hi,
> 
> We've been talking for a while about adding index support to
> Scrollkeeper. It looks like we have the people to work on it now, Mary
> Dwyer (from Sun), John Fleck and Sasha most of the time. mary would
> probably do the development work.
> 
> I frame up my idea about how should it be done and we can discuss it.
> 
> I dont know much about index tags in DocBook/SGML so I am not sure how
> we implement this. I think there are two ways of indexing, one that just
> defines the index term and an "anchor" to where it is discussed (or
> maybe more anchors?) and the other holds the index definition itself.
> 
> Anyway I think the way it should be done is that we take the SGML file
> and extract the index info from it into a very simple xml file. Ideally
> this should contain only the index strings and the "anchor" in the text
> that is associated with them. I dont know if the SGML index tags support
> this aproach.
> 
> On the technical side I think we should use a SAX interface rather then
> a DOM tree as very little info from the SGML docs is actually used.
> 
> The index extractor should be a new binary or part of the Scrollkeeper
> library if we manage to turn SK to a library.
> 
> This is a very short description, but I think it contains the main
> issues.
> 
> What do you think about it?
> 
> Laszlo
>

Re: [Scrollkeeper-devel] Index support

[John F. <jf...@sw...>]

Hi,

I've mocked up a short DocBook document below with what I hope are
relatively complete examples of the use of indexterm tags as an
example to get our discussion of indexing in ScrollKeeper started.

From a functional point of view, I'd like the system to be capable of
two things:

1) Providing the data necessary to the help system browser to generate a
system-wide index of all the documents registered with ScrollKeeper

2) Providing the data necessary for the browser to generate a
document-specific appendix for a particular document

An optional third capability I'd like to consider is a middle ground
that is likely implicit in 1) above, the capability to generate a
system-wide index for all the documentation in a given SK category.

On Tue, Mar 13, 2001 at 05:21:05PM +0000, Laszlo Kovacs wrote:
>
> Anyway I think the way it should be done is that we take the SGML file
> and extract the index info from it into a very simple xml file. Ideally
> this should contain only the index strings and the "anchor" in the text
> that is associated with them.

This seems reasonable. I see something along these lines, with a
series of "terms" - the terms to be listed in the index - with an
"instance" for each occurence of the term as an indexentry in a
document. Each instance would then record the necessary anchor
information. To be most useful, we should record both the id of
the indexterm itself and the id of the enclosing sect. That would
allow flexibility for the help browser folks.

I leave the specifics of the xml data structure to the wizards in the
group, but conceptually I'm thinking of something along these lines:

***************************************

<indexentry term="term"> (this would be based on the primary term in
			  an indexterm)

	<instance>
		<secondary>secondaryterm</secondary> (optional)

		<doc>/pathtofile/filename.sgml</doc> (the location of
						      the file)

		<category>category</category>	     (the omf category
						      of the doc)

		<sectid>sectid</sectid>              (the id of the
						      section in which
						      the indexterm is
						      located)

		<indextermid>idid</indextermid>      (the id of the
						      indexterm
						      itself)
	
		<indextermrangeend>anotherid</indextermrangeend> (the
						      id for the end
						      of the indexterm
						      range if used)

		<seealso>otherterm</seealso>         (a pointer to
						      another term - optional)
	</instance>

	<instance>
	    ............. another instance .............
            ............. possibly in the same doc .....
	    ............. or a different one ...........
	</instance>

</indexentry> 

*****************************************

Some questions to consider:

- how should we handle locale? I expect we should have a separate xml
index file for each locale, based on the language code in the omf.

- DocBook indexterm also supports attributes "zone," "significance",
"scope" and "pagenum". How should we treat those?

A sample doc:
***********************************************************
<!DOCTYPE article PUBLIC "-//OASIS//DTD DocBook V3.1//EN" [
]>
<article id="index">

  <sect1 id="sect1-01">
    <title>Indexterm</title>
    <para>
      <indexterm id="idx-id1">
	<primary>indexterm</primary>
      </indexterm>
      This paragraph contains some content that will be indexed using an
      ordinary single-point indexterm.</para>
  </sect1>

  <sect1 id="sect1-02">
    <title>Range</title>
    <para>
      <indexterm id="idx-id2" class="startofrange">
	<primary>indexterm</primary>
	<secondary>range</secondary>
      </indexterm>
      This paragraph contains some content that will be indexed using an
      indexterm range.
      <indexterm startref="idx-id2" class="endofrange">
	
    </para>
  </sect1>


  <sect1 id="sect1-03">
    <title>Seealso</title>
    <para>
      <indexterm id="idx-id3">
	<primary>seealso</primary>
	<seealso>indexterm</seealso>
      </indexterm>
      This paragraph contains content regarding the seealso tag, with a seealso
      reference to indexterm in the index.
    </para>

    <sect2 id="sect2-01">
      <title>Indexterm in nested sect2</title>
      <para>
      <indexterm id="idx-id4">
	<primary>nested</primary>
      </indexterm>
	Here is some text in a sect2 that will be indexed.
      </para>
    </sect2>

  </sect1>

</article>
*****************************************************

Cheers,
-- 
John Fleck
jf...@sw... (h)
jf...@ab... (w)
http://www.abqjournal.com/scitech/

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@su...>]

John Fleck wrote:
> 
> Hi,
> 
> I've mocked up a short DocBook document below with what I hope are
> relatively complete examples of the use of indexterm tags as an
> example to get our discussion of indexing in ScrollKeeper started.
> 
> From a functional point of view, I'd like the system to be capable of
> two things:
> 
> 1) Providing the data necessary to the help system browser to generate a
> system-wide index of all the documents registered with ScrollKeeper
> 
> 2) Providing the data necessary for the browser to generate a
> document-specific appendix for a particular document

Could you detail the appendix a bit please? I am not sure what it is
exactly. Apart from this I think Scrollkeeper should be able to extract
index from one doc and offer that two the user. We can build a
system-wide index on that as offered by Scrollkeeper or we can leave it
to the browser to do this.
 
> An optional third capability I'd like to consider is a middle ground
> that is likely implicit in 1) above, the capability to generate a
`> system-wide index for all the documentation in a given SK category.
This is simple if we store separately the index info of each doc.

> This seems reasonable. I see something along these lines, with a
> series of "terms" - the terms to be listed in the index - with an
> "instance" for each occurence of the term as an indexentry in a
> document. Each instance would then record the necessary anchor
> information. To be most useful, we should record both the id of
> the indexterm itself and the id of the enclosing sect. That would
> allow flexibility for the help browser folks.
Storing the enclosing sect also might be a good idea.

> 
> I leave the specifics of the xml data structure to the wizards in the
> group, but conceptually I'm thinking of something along these lines:
> 
> ***************************************
> 
> <indexentry term="term"> (this would be based on the primary term in
>                           an indexterm)
> 
>         <instance>
>                 <secondary>secondaryterm</secondary> (optional)
> 
>                 <doc>/pathtofile/filename.sgml</doc> (the location of
>                                                       the file)
> 
>                 <category>category</category>        (the omf category
>                                                       of the doc)
> 
>                 <sectid>sectid</sectid>              (the id of the
>                                                       section in which
>                                                       the indexterm is
>                                                       located)
> 
>                 <indextermid>idid</indextermid>      (the id of the
>                                                       indexterm
>                                                       itself)
> 
>                 <indextermrangeend>anotherid</indextermrangeend> (the
>                                                       id for the end
>                                                       of the indexterm
>                                                       range if used)
> 
>                 <seealso>otherterm</seealso>         (a pointer to
>                                                       another term - optional)
>         </instance>
> 
>         <instance>
>             ............. another instance .............
>             ............. possibly in the same doc .....
>             ............. or a different one ...........
>         </instance>
> 
> </indexentry>

This sounds resonable with the addition that we are looking into the
implementation of a doc id that would be the same for a doc and all its
translations, so the actual information about the doc stored here might
be a bit different.

> 
> *****************************************
> 
> Some questions to consider:
> 
> - how should we handle locale? I expect we should have a separate xml
> index file for each locale, based on the language code in the omf.
Yes. This should be pretty similar to the TOC implementation. Right now
Scrollkeeper assigns a unique id to every installed doc and the TOC of
the doc is stored in the TOC subdirectory of the scrollkeeper database
directory in a file that has the ID as its name.
 
> - DocBook indexterm also supports attributes "zone," "significance",
> "scope" and "pagenum". How should we treat those?
I have no idea, this is up to people who know more about SGML in
general.

Finally dont be shy to cut out everything from your mails quoted texts
that is not related as we already managed to grow emails to unbearable
sizes at previous design related discussions.

Laszlo

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@su...>]

> My thinking here was that SK would put the index data into an xml data
> structure that was browser-independent, then people's browsers would
> use that data to generate a displayable version of the
> document-specific index, showing up as an appendix to the document. I
> have some ideas about how we might do that for
> GNOME, but I assume others might have other ideas for other platforms,
> and my thinking was that the SK data structure would be a
> platform-neutral xml structure we all could draw on.
This is precisely how it should work.
 
> As an alternative, I suppose, we could have SK just generate a DocBook
> index document for the doc at install time and stick it in the same
> directory as the doc.
I think we need both. We need index associated with every doc and we
also need joint index of several docs or all the docs from the system.
The basis should obviously be separate index for each doc and then joint
indexes can be built on that easily.
 
> So you've have the large systemwide xml data structure containing all
> index data, and a separate index.sgml (or index.xml) that gets stuck
> in the install directory with the doc. Would that work?
Something like this will work definitely and this is how I think it
should be done.

Laszlo

Re: [Scrollkeeper-devel] Index support

[Dan M. <da...@ea...>]

Before we get too far into technical details of how to implement indexing,
I'd like to discuss the general ideas for what we want to achieve.  Here I
think we should start from the perspective of the end-user sitting in
front of a help browser.  The help browser I am thinking of is the GNOME
help browser or perhaps the KDE help browser.  However, what we design
must be able to work just as well for a general Linux help browser, *nix
help browser, or web-based help browser for project Foo or organization
Bar.

Here are a few of my thoughts on what sort of functionality a user may
want.

1) "Basic Document Index" 

For any document, a user may want an index.  Here I mean "index" as what
one normally finds at the back of a book.  It is a list of terms, often
with secondary or even tertiary headings and "see also" items.  These
terms link to concepts and/or words in the body of the document.  The
purpose is to allow a user who is looking for information on a particular
concept to quickly find the part(s) of the document which discuss this
concept.

DocBook provides tags to markup terms to appear in an index.  When the
document is processed, such as when one converts the SGML to HTML or PS,
it generates the index from the many marked up terms throughout the
document.  For HTML, a link is provided from the index to the correct part
of the document.  For PS, I believe the page number is listed.

This is a very useful level of indexing which is already possible without
ScrollKeeper.  The only requirements are that an experienced indexer
manually add the indexing markup and that the document processor knows how
to generate the index.  (Note this requires a fair amount of expertise and
time on the part of the indexer, but I believe the technology is complete
for DocBook/SGML documents rendered into the standard target formats.)

2) "Help System Aware Document Index"

To increase usability, the help browser is aware of the document index
from #1, rather than merely displaying the content of the index.  This
allows the user to do things like:

a) browse the index in one window or portion of the help browser while
viewing the document in another

b) more quickly navigate and search the index using an index browser built
into the help system

c) search multiple documents to identify which one discusses a certain
topic, or even create compound indexes.

d) (many other things which are possible, but which probably should be
done in ScrollKeeper instead of the help browser.  I discuss some of them 
below.)

I am mainly interested in (a) and (b) above, since I think (c) is probably
something which should be in (d).

3) "System-wide Index"

This is an index of all documents on the system, obtained my merging all
of the indexes to all of the documents (of a given locale and excluding
duplicates of a given document) on the system.

A system-wide index may be valuable for searching for more obscure things,
like "beowulf".  However, putting in "cd-rom" or "printer" will likely
return so many items it may be completely useless to the user.

4) "Group Indexes"

This is an index of a certain set of documents which may not span the
entire set of documents.  For example, it may be an index for all GNOME
documents, or all KDE documents, or all documents written by John.  I
believe this will be the most valuable index type.

In fact, for GNOME I see the need for at least two different groups.  One
for the core desktop, and another for all GNOME documents.  Similar to the
example above, it would be nice to enter "printer" and get only a few hits
about configuring and using printers, not the "how to print from
application foo" for every application (or every GNOME application) on
your computer.  So generally I would expect a help browser to request
indexes from ScrollKeeper for multiple groups of documents.

--

A few thoughts on technical requirements:

1) Index Targets

We need to be able to parse the DocBook/SGML documents and extract
their indexing terms and anchors.  Identifying the indexing terms should
be straightforward.  The other half is predicting the anchors in the
document.  This is another example of where we require our document
processors to behave in a uniform way.  For example, if we have two
DocBook to HTML converters, we need them both to label their anchors the
same way.  

Also, I think we may soon have a good way to render DocBook/XML documents
directly.  I'm not sure how we would specify the targets in this case.  
(Could anybody provide some information on how this would work?)

We will need to resolve this issue to achieve any of the 4 functional
points listed above except #1.

2) General Database and API

We should make the index database as general as possible.  Ideally, for
each occurance of each indexing term, we associate all of the OMF metadata
of the document.  This allows the help browser to make arbitrary queries,
from requesting an index for a single document, to an index of the whole
system, to an index of an arbitrary list of documents, to an index with
certain metadata (eg. documents with a <subject> of "Core GNOME
Documents").

This would mean we want to select a free database which ScrollKeeper will
use to manage all of the indexing data.  The exported ScrollKeeper API for
returning various indexes would probably be a rather thin wrapper around
the database.  It also means that ScrollKeeper will leave the innocent
world of scripts and command-line programs which point to XML files and
at least the indexing portion will need to be a proper library which
browsers link to.

This approach is probably a fair amount more work than using XML files as
we have in the past.  It will also require more up-front investment in
terms of design and expertise.  However, I think we need the speed and
flexibility of a real database in order to provide some of the most
valuable functionality: The ability to produce indexes on specific groups
of documents, specified by their metadata.

Dan

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@Su...>]

> Here are a few of my thoughts on what sort of functionality a user may
> want.
> 
> 1) "Basic Document Index"
> 
> For any document, a user may want an index.  Here I mean "index" as what
> one normally finds at the back of a book.  It is a list of terms, often
> with secondary or even tertiary headings and "see also" items.  These
> terms link to concepts and/or words in the body of the document.  The
> purpose is to allow a user who is looking for information on a particular
> concept to quickly find the part(s) of the document which discuss this
> concept.
> 
> DocBook provides tags to markup terms to appear in an index.  When the
> document is processed, such as when one converts the SGML to HTML or PS,
> it generates the index from the many marked up terms throughout the
> document.  For HTML, a link is provided from the index to the correct part
> of the document.  For PS, I believe the page number is listed.
> 
> This is a very useful level of indexing which is already possible without
> ScrollKeeper.  The only requirements are that an experienced indexer
> manually add the indexing markup and that the document processor knows how
> to generate the index.  (Note this requires a fair amount of expertise and
> time on the part of the indexer, but I believe the technology is complete
> for DocBook/SGML documents rendered into the standard target formats.)
> 
> 2) "Help System Aware Document Index"
> 
> To increase usability, the help browser is aware of the document index
> from #1, rather than merely displaying the content of the index.  This
> allows the user to do things like:
> 
> a) browse the index in one window or portion of the help browser while
> viewing the document in another
> 
> b) more quickly navigate and search the index using an index browser built
> into the help system
> 
> c) search multiple documents to identify which one discusses a certain
> topic, or even create compound indexes.
> 
> d) (many other things which are possible, but which probably should be
> done in ScrollKeeper instead of the help browser.  I discuss some of them
> below.)
> 
> I am mainly interested in (a) and (b) above, since I think (c) is probably
> something which should be in (d).
> 
> 3) "System-wide Index"
> 
> This is an index of all documents on the system, obtained my merging all
> of the indexes to all of the documents (of a given locale and excluding
> duplicates of a given document) on the system.
> 
> A system-wide index may be valuable for searching for more obscure things,
> like "beowulf".  However, putting in "cd-rom" or "printer" will likely
> return so many items it may be completely useless to the user.
> 
> 4) "Group Indexes"
> 
> This is an index of a certain set of documents which may not span the
> entire set of documents.  For example, it may be an index for all GNOME
> documents, or all KDE documents, or all documents written by John.  I
> believe this will be the most valuable index type.
> 
> In fact, for GNOME I see the need for at least two different groups.  One
> for the core desktop, and another for all GNOME documents.  Similar to the
> example above, it would be nice to enter "printer" and get only a few hits
> about configuring and using printers, not the "how to print from
> application foo" for every application (or every GNOME application) on
> your computer.  So generally I would expect a help browser to request
> indexes from ScrollKeeper for multiple groups of documents.

Sounds good.

> 
> A few thoughts on technical requirements:
> 
> 1) Index Targets
> 
> We need to be able to parse the DocBook/SGML documents and extract
> their indexing terms and anchors.  Identifying the indexing terms should
> be straightforward.  The other half is predicting the anchors in the
> document.  This is another example of where we require our document
> processors to behave in a uniform way.  For example, if we have two
> DocBook to HTML converters, we need them both to label their anchors the
> same way.
> 
> Also, I think we may soon have a good way to render DocBook/XML documents
> directly.  I'm not sure how we would specify the targets in this case.
> (Could anybody provide some information on how this would work?)
> 
> We will need to resolve this issue to achieve any of the 4 functional
> points listed above except #1.

I guess the way how the browser is able or not to jump to the index ID
provided is up to the browser, not Scrollkeeper. We just have to be sure
that we push the usage of those index terms from SGML that facilitate
the implementation of the jump to the right index content.

> 
> 2) General Database and API
> 
> We should make the index database as general as possible.  Ideally, for
> each occurance of each indexing term, we associate all of the OMF metadata
> of the document.  This allows the help browser to make arbitrary queries,
> from requesting an index for a single document, to an index of the whole
> system, to an index of an arbitrary list of documents, to an index with
> certain metadata (eg. documents with a <subject> of "Core GNOME
> Documents").
> 
> This would mean we want to select a free database which ScrollKeeper will
> use to manage all of the indexing data.  The exported ScrollKeeper API for
> returning various indexes would probably be a rather thin wrapper around
> the database.  It also means that ScrollKeeper will leave the innocent
> world of scripts and command-line programs which point to XML files and
> at least the indexing portion will need to be a proper library which
> browsers link to.
> 
> This approach is probably a fair amount more work than using XML files as
> we have in the past.  It will also require more up-front investment in
> terms of design and expertise.  However, I think we need the speed and
> flexibility of a real database in order to provide some of the most
> valuable functionality: The ability to produce indexes on specific groups
> of documents, specified by their metadata.

I think the database is not feasible in the near future. I think the two
most obvious queries are searching for a term in one doc's index and
searching in a group of docs' indexes where the docs are grouped
according to their metadata subject field (the same that is used to add
them to the right place in the category tree). I am sure people can come
up with loads of useful queries, but I think these will be far the most
common. As we have the docs in the category tree ordered by subject,
grouping according to subject is easy. Merging a number of indexes into
a temporary one will not be very difficult. And this is all we need.

The problem with the free database approach is that we dont know which
one is available on the most OS-es, we certainly know that none right
now on Solaris. While this might be sorted in the future, it is
certainly not very close. The approach I described above can be
implemented XML based in 2-3 months and it will work with a not very
large amount of docs. A database approach would take much longer. I dont
exclude the possibility of a parallel XML/database approach or moving
totally to a database though.

Laszlo