scrollkeeper-devel

[Scrollkeeper-devel] structure of extracted index page

[Mary D. <Mar...@Su...>]

hi

I'd appreciate some feedback/suggestions on the structure of the index 
scrollkeeper will create from a document.

To aid explanation, consider a document including the following index markups:

<indexterm id="idx-a1">
  <primary>Apple</primary><secondary>Big </secondary><tertiary>Green</tertiary>
</indexterm>

<indexterm zone="a1"><primary>Orange</primary><secondary>Medium></secondary>
</indexterm>

<indexterm id="idx-a2" class=startofrange>
  <primary>Banana</primary><secondary>Small</secondary>
</indexterm>
<indexterm startref="idx-a2" class=endofrange>




The Example below is an excerpt from the extracted index .
 
1. The tags <indexdoc> </indexdoc> indicate beginning and end of document
2. The index entry is indicated by the tags <indexentry linkid="id"> 
</indexentry>

I do not know how to handle See and See Also references (as they are not 
associatied with an id) - any suggestions?


Example:

<indexdoc>
   <indexentry linkid="idx-a1">Apple, Big, Green
   </indexentry>
   <indexentry linkid="idx-a2">Banana, Small
   </indexentry>
   <indexentry linkid="a1">Orange, Medium
   </indexentry
      
   etc. .......
   
</indexdoc>   



TIA
Mary




~ I speak for myself, not for my employer ~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Mary Dwyer
Desktop Applications & Middleware Grp
Sun Microsystems Ireland
Tel: +353-1-8199222 (xt 19222)
Fax: +353-1-8199078
email: mar...@ir...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Re: [Scrollkeeper-devel] structure of extracted index page

[Dan M. <da...@ea...>]

Hi Mary,

I suspect you know more about indexing with DocBook than almost anybody
else on these lists.  I'll give a shot at an answer, and you can correct
me where I go wrong ;)

My understanding is that id's are optional on all the indexing tags of
interest here: indexterm, see, seealso (and maybe others...)

Do you know what collateindex.pl does with indexterms which do not have
id's?  I tried to make a small test document which generates a nice index
but wasn't successful within just a few minutes.  Do you have a test
document handy we can play with?  I am guessing that it can generate an
index independent of whether the indexterms have id's.

It is certainly convenient if all the indexterms have id's, as it is
easier to link to them from the index.  This is very similar to how the
TOC links to section id's though.  Jade deals with sections without id's
by assigning id's to them.  It can do this because it is generating both
the anchors and the links in the same output.

Suppose we continued to use on-the-fly conversion from SGML to HTML:

The difficulty with ScrollKeeper creating an index off of an SGML
document, or even a TOC off an SGML document, which doesn't have id's is
that the generated index or TOC has to predict the id's (ie. anchors)
which will be assigned to those sections or indexterms by the converter at
run time.  If we know how this assignment will be done, we are ok.  
Otherwise, we must require that all sections and indexterms (and <see> and
<seealso>) have id's.  This is what we are doing now in the GDP, but is
not really a great solution since we are making further restrictions on
top of DocBook.  Thus SK would not work with just any DocBook doc, but a
certain subset of all DocBook docs.  So the better solution is to come up
with a scheme which will assign id's in a predictable way.  This method
would be used by ScrollKeeper during the index creation and during
gnome-db2html2/gnome-db2html3 during display.  The downside to this is
that ScrollKeeper would need to know in advance which display system will
be used.  So long as GNOME and KDE follows the system used by
collateindex.pl, we should not have any problems.

Laszlo - Does this all sound correct based on your experience?  How do you
handle sections without id's in ScrollKeeper's TOC extraction?  Do you
ignore those sections or id them in the same way as db2html?

The other possibility is that instead of trying to refer to an anchor in
the generated HTML, we try to refer to the position in the XML document.  
I really don't know how this would work exactly, since I am not very
familiar with libxml, but it may be possible.  (DV?)

Dan



On Wed, 25 Apr 2001, Mary Dwyer wrote:

> hi
> 
> I'd appreciate some feedback/suggestions on the structure of the index 
> scrollkeeper will create from a document.
> 
> To aid explanation, consider a document including the following index markups:
> 
> <indexterm id="idx-a1">
>   <primary>Apple</primary><secondary>Big </secondary><tertiary>Green</tertiary>
> </indexterm>
> 
> <indexterm zone="a1"><primary>Orange</primary><secondary>Medium></secondary>
> </indexterm>
> 
> <indexterm id="idx-a2" class=startofrange>
>   <primary>Banana</primary><secondary>Small</secondary>
> </indexterm>
> <indexterm startref="idx-a2" class=endofrange>
> 
> 
> 
> 
> The Example below is an excerpt from the extracted index .
>  
> 1. The tags <indexdoc> </indexdoc> indicate beginning and end of document
> 2. The index entry is indicated by the tags <indexentry linkid="id"> 
> </indexentry>
> 
> I do not know how to handle See and See Also references (as they are not 
> associatied with an id) - any suggestions?
> 
> 
> Example:
> 
> <indexdoc>
>    <indexentry linkid="idx-a1">Apple, Big, Green
>    </indexentry>
>    <indexentry linkid="idx-a2">Banana, Small
>    </indexentry>
>    <indexentry linkid="a1">Orange, Medium
>    </indexentry
>       
>    etc. .......
>    
> </indexdoc>   
> 
> 
> 
> TIA
> Mary
> 
> 
> 
> 
> ~ I speak for myself, not for my employer ~
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Mary Dwyer
> Desktop Applications & Middleware Grp
> Sun Microsystems Ireland
> Tel: +353-1-8199222 (xt 19222)
> Fax: +353-1-8199078
> email: mar...@ir...
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> 
> 
> _______________________________________________
> Scrollkeeper-devel mailing list
> Scr...@li...
> http://lists.sourceforge.net/lists/listinfo/scrollkeeper-devel
> 

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Wed, Apr 25, 2001 at 10:44:30PM -0500, Dan Mueth wrote:
> The other possibility is that instead of trying to refer to an anchor in
> the generated HTML, we try to refer to the position in the XML document.  
> I really don't know how this would work exactly, since I am not very
> familiar with libxml, but it may be possible.  (DV?)

  I'm afraid I didn't follow the discussion here (sorry !), the best
way to get a technical  answer from me is to give me a practical example
(what's your input, how it's processed, what's the result, why it fails),
and then I can use what I know both from the specs and the code to get
this answer precisely and quickly,

Daniel

-- 
Daniel Veillard      | Red Hat Network http://redhat.com/products/network/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[Dan M. <da...@ea...>]

On Thu, 26 Apr 2001, Daniel Veillard wrote:

> On Wed, Apr 25, 2001 at 10:44:30PM -0500, Dan Mueth wrote:
> > The other possibility is that instead of trying to refer to an anchor in
> > the generated HTML, we try to refer to the position in the XML document.  
> > I really don't know how this would work exactly, since I am not very
> > familiar with libxml, but it may be possible.  (DV?)
> 
>   I'm afraid I didn't follow the discussion here (sorry !), the best
> way to get a technical  answer from me is to give me a practical example
> (what's your input, how it's processed, what's the result, why it fails),
> and then I can use what I know both from the specs and the code to get
> this answer precisely and quickly,

Ok.  Let me give this a shot...

<disclaimer>
The DTD represented in this email is fictional and any similarity
with a real DTD is purely coincidental.
</disclaimer>

<sect1>
  <para>
    This is a sentence.
  </para>
  <indexterm>
    <primary>
      Sentences
    </primary>
  </indexterm>
  <para>
    This is another sentence.
  </para>
</sect1>

When this is converted to HTML, we will get an index at the end of the
document which has a link from an item called "Sentences" to the location
of the indexterm element above: between the two paragraphs.  Populating
our document with indexterms yields a helpful index at the end of the
document :)

We would like ScrollKeeper to keep an XML data file describing the
index.  It should list all of the index terms and where they link into the
document.

The thing I am not sure about is how we "anchor" the links into the
document.  If the indexterm had a unique id attribute, we could use
that.  But our DTD does not require the id attribute be used.

Is there a nice way we could have an XML representation of the index which
somehow specifies the anchors for the index term links so that a browser
(such as the help browser in Nautilus) can link from index terms to
locations in the XML document?

I hope this was clearer.

Dan

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Thu, Apr 26, 2001 at 02:24:47AM -0500, Dan Mueth wrote:
> 
> On Thu, 26 Apr 2001, Daniel Veillard wrote:
> 
> > On Wed, Apr 25, 2001 at 10:44:30PM -0500, Dan Mueth wrote:
> > > The other possibility is that instead of trying to refer to an anchor in
> > > the generated HTML, we try to refer to the position in the XML document.  
> > > I really don't know how this would work exactly, since I am not very
> > > familiar with libxml, but it may be possible.  (DV?)
> > 
> >   I'm afraid I didn't follow the discussion here (sorry !), the best
> > way to get a technical  answer from me is to give me a practical example
> > (what's your input, how it's processed, what's the result, why it fails),
> > and then I can use what I know both from the specs and the code to get
> > this answer precisely and quickly,
> 
> Ok.  Let me give this a shot...
> 
> <disclaimer>
> The DTD represented in this email is fictional and any similarity
> with a real DTD is purely coincidental.
> </disclaimer>
> 
> <sect1>
>   <para>
>     This is a sentence.
>   </para>
>   <indexterm>
>     <primary>
>       Sentences
>     </primary>
>   </indexterm>
>   <para>
>     This is another sentence.
>   </para>
> </sect1>
> 
> When this is converted to HTML, we will get an index at the end of the
> document which has a link from an item called "Sentences" to the location
> of the indexterm element above: between the two paragraphs.  Populating
> our document with indexterms yields a helpful index at the end of the
> document :)
> 
> We would like ScrollKeeper to keep an XML data file describing the
> index.  It should list all of the index terms and where they link into the
> document.
> 
> The thing I am not sure about is how we "anchor" the links into the
> document.  If the indexterm had a unique id attribute, we could use
> that.  But our DTD does not require the id attribute be used.
> 
> Is there a nice way we could have an XML representation of the index which
> somehow specifies the anchors for the index term links so that a browser
> (such as the help browser in Nautilus) can link from index terms to
> locations in the XML document?

  If you want to point into the XML then you need to use XPointer,
if you had an ID on the element, say "sentencedef", then the simple
way to addres the subpart is

     #sentencedef

if you don't have such an ID in this case you can use the structured
access method of XPointer in this case one such pointer could be:

     #xpointer(/sect1[1]/indexterm[1]/primary[1])

Libxml has an XPointer implementation

orchis:~/XML -> cat tst.xml
<sect1>
  <para>
    This is a sentence.
  </para>
  <indexterm>
    <primary>
      Sentences
    </primary>
  </indexterm>
  <para>
    This is another sentence.
  </para>
</sect1>

orchis:~/XML -> ./testXPath -xptr -i tst.xml "xpointer(/sect1[1]/indexterm[1]/primary[1])"
Object is a Node Set :
Set contains 1 nodes:
1  ELEMENT primary
orchis:~/XML -> 

  XPointer is not (yet) widely deployed, I just happen to be the co-chair
of the working group defining it at W3C, you can get the spec at
    http://www.w3.org/TR/xptr

and I can of course answer questions about it.

  In practice, getting IDs is better because it's more resilient to changes
another improvement would be to have one term per primary tag like this
<primary>sentence</primary>
then a very resilent XPointer would be:

  #xpointer(//indexterm/primary[. = "sentence"])

Basically it instructs to search all indexterm in the document, then
look for primary children and extract the ones where the content is
 "sentence"

  All those queries are actually XPath expressions (used in XSLT too),
you can get more informations at :
   - a generic presentation on XPointer
     http://daniel.veillard.com/Talks/9912XPointer/Overview.html

   - the W3C page on XPointer (at the bottom of the page)
     http://www.w3.org/XML/Linking.html

   - XPointer libxml interfaces:
    http://xmlsoft.org/html/libxml-xpointer.html

Daniel

-- 
Daniel Veillard      | Red Hat Network http://redhat.com/products/network/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[<las...@Su...>]

> Laszlo - Does this all sound correct based on your experience?  How do you
> handle sections without id's in ScrollKeeper's TOC extraction?  Do you
> ignore those sections or id them in the same way as db2html?
Yes, sections with no IDs are ignored  during TOC extraction.

Laszlo

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Thu, Apr 26, 2001 at 10:29:42AM +0100, L=E1szl=F3 Kov=E1cs wrote:
> > Laszlo - Does this all sound correct based on your experience?  How d=
o you
> > handle sections without id's in ScrollKeeper's TOC extraction?  Do yo=
u
> > ignore those sections or id them in the same way as db2html?
> Yes, sections with no IDs are ignored  during TOC extraction.

  Hum, this should probably be improved ...
C.f. my other mail for a possible technical solution.

Daniel

--=20
Daniel Veillard      | Red Hat Network http://redhat.com/products/network=
/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[<las...@Su...>]

Daniel Veillard wrote:
> =

> On Thu, Apr 26, 2001 at 10:29:42AM +0100, L=E1szl=F3 Kov=E1cs wrote:
> > > Laszlo - Does this all sound correct based on your experience?  How=
 do you
> > > handle sections without id's in ScrollKeeper's TOC extraction?  Do =
you
> > > ignore those sections or id them in the same way as db2html?
> > Yes, sections with no IDs are ignored  during TOC extraction.
> =

>   Hum, this should probably be improved ...
> C.f. my other mail for a possible technical solution.
> =

> Daniel

Are you talking about the XPointer email? I dont understand how that
helps us here. Our problem is that if a section does not have a unique
id then Scrollkeeper and gnome-db2html[2|3] can jump there only if they
generate an id to this section which is the same in both Scrollkeeper
and the convertor. I am not sure how could XPointer help us in this
case. I think the examples you gave described how to identify one or
more subparts of the xml doc. But as far as I understand we still need
something to uniquely identify the subpart we are going for to use the
XPointer mechanism and we dont have uniqueness assured. Unless I
misunderstand something badly which is possible as I didnt spend a huge
amount of time with XPointer yet.

Laszlo

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Thu, Apr 26, 2001 at 12:13:25PM +0100, L=E1szl=F3 Kov=E1cs wrote:
> > > Yes, sections with no IDs are ignored  during TOC extraction.
> >=20
> >   Hum, this should probably be improved ...

  That is sure.

> > C.f. my other mail for a possible technical solution.

  Now whether XPointer can help w.r.t. the addressability then it really
depends what need to be addressed.

> Are you talking about the XPointer email? I dont understand how that
> helps us here. Our problem is that if a section does not have a unique
> id then Scrollkeeper and gnome-db2html[2|3] can jump there only if they
> generate an id to this section which is the same in both Scrollkeeper
> and the convertor.

  What do you mean by jump here:
    - if the here is an XML document, then I assume the document is
      handled by libxml and hence XPointer can be used
    - if the here is an HTML document, then XSLT has a=20
      generate-id() function which can be used to generate a unique ID
      for this element, and pointing is also possible using the existing
      #name framework.

Did i missed something ?

Daniel

--=20
Daniel Veillard      | Red Hat Network http://redhat.com/products/network=
/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[<las...@Su...>]

Daniel Veillard wrote:
> 
>   What do you mean by jump here:
>     - if the here is an XML document, then I assume the document is
>       handled by libxml and hence XPointer can be used
>     - if the here is an HTML document, then XSLT has a
>       generate-id() function which can be used to generate a unique ID
>       for this element, and pointing is also possible using the existing
>       #name framework.
> 
> Did i missed something ?

Yes, we have two systems parsing the same document (scrollkeeper and
gnome-db2html[2|3]) and they can cooperate only if they generate the
same "unique" id. I think this is our problem as this seems to be
impossible.

Laszlo

Re: [Scrollkeeper-devel] structure of extracted index page

[<mv...@df...>]

> Yes, we have two systems parsing the same document (scrollkeeper and
> gnome-db2html[2|3]) and they can cooperate only if they generate the
> same "unique" id. I think this is our problem as this seems to be
> impossible.

Right, this is important, and needs to be solved. I guess scrollkeeper
would need to guess the result of gnome-db2html[2|3]? Can we make the
output from gnome-db2html3 be dependable?

Martin

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Thu, Apr 26, 2001 at 01:47:47PM +0100, L=E1szl=F3 Kov=E1cs wrote:
> > Did i missed something ?
>=20
> Yes, we have two systems parsing the same document (scrollkeeper and
> gnome-db2html[2|3]) and they can cooperate only if they generate the
> same "unique" id. I think this is our problem as this seems to be
> impossible.

  Use some property of the input element to generate this ID.
Example:

<doc>
     <row/>
     <row/>
     <row>
	  <para/>
          <colum/>
          <colum/>
          <colum/>
          <colum>
	      <element_without_ID/>
	  </colum>
...

Associate it the following ID: SK_3_5_1 (child sequence when going from t=
he
root).
Both programs would compute the same ID since it's based on the structure
of the input document (shared). This is based on the child sequence mecha=
nism
of XPointer.

problem: it doesn't resist well to changes in the input, but in this case
	 I assume it's not a problem.

Daniel


--=20
Daniel Veillard      | Red Hat Network http://redhat.com/products/network=
/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[<las...@Su...>]

Daniel Veillard wrote:

>   Use some property of the input element to generate this ID.
> Example:
> 
> <doc>
>      <row/>
>      <row/>
>      <row>
>           <para/>
>           <colum/>
>           <colum/>
>           <colum/>
>           <colum>
>               <element_without_ID/>
>           </colum>
> ...
> 
> Associate it the following ID: SK_3_5_1 (child sequence when going from the
> root).
> Both programs would compute the same ID since it's based on the structure
> of the input document (shared). This is based on the child sequence mechanism
> of XPointer.
> 
> problem: it doesn't resist well to changes in the input, but in this case
>          I assume it's not a problem.
> 
> Daniel
> 

This could be a good idea. But it might make Scrollkeeper Gnome
dependent. Unless we come up with a common stylesheet that Scrollkeeper
exports and suggests to be used by whatever rendering mechanism is
implemented in the browser. It still might be a too tight connection
though.

Laszlo

Re: [Scrollkeeper-devel] structure of extracted index page

[Daniel V. <vei...@re...>]

On Thu, Apr 26, 2001 at 02:03:57PM +0100, L=E1szl=F3 Kov=E1cs wrote:
> This could be a good idea. But it might make Scrollkeeper Gnome
> dependent.

  I don't understand why, this computation has nothing Gnome specific
int it !

  I really don't understand your framework, please detail all the pieces
because it's getting frustrating ... All I understand is that at the
beginning there is a single docbook (XML) file and then multiple HTML=20
views are generated, please exaplin the different paths of the processing
(unless you already wrote it in some pages I can read).

Daniel

--=20
Daniel Veillard      | Red Hat Network http://redhat.com/products/network=
/
vei...@re...  | libxml Gnome XML XSLT toolkit  http://xmlsoft.org/
http://veillard.com/ | Rpmfind RPM search engine http://rpmfind.net/

Re: [Scrollkeeper-devel] structure of extracted index page

[<las...@Su...>]

>   I don't understand why, this computation has nothing Gnome specific
> int it !
:-) Sorry Daniel. I was just thinking loudly. You are right, it is not
Gnome specific, I was wrong. It will need any Scrollkeeper dependant
browser to generate the IDs in the same way as Scrollkeeper does it.
Which might be a problem if the browser is already implemented and it is
just integrated with Scrollkeeper. 

Thanks for the help,
 
Laszlo

Re: [Scrollkeeper-devel] structure of extracted index page

[Dan M. <da...@ea...>]

[removing gnome-doc-list from cc list since this email is getting into the
internal guts of SK]

On Wed, 25 Apr 2001, Mary Dwyer wrote:

> Example:
> 
> <indexdoc>
>    <indexentry linkid="idx-a1">Apple, Big, Green
>    </indexentry>
>    <indexentry linkid="idx-a2">Banana, Small
>    </indexentry>
>    <indexentry linkid="a1">Orange, Medium
>    </indexentry
>       
>    etc. .......
>    
> </indexdoc>   

There is nesting information which is left out.  For example, if you also
have "Apple, Big, Blue", the markup above does not reveal that your index
looks like:
  Apple
      Big
          Green  *
          Blue   *
  Banana
      Small      *
  Orange
      Medium     *

I think it makes sense to put this into the XML file, since this is the
way the data will typically be displayed and processed.  If SK does the
work of sorting this out then things will be faster and easier for the
help browsers.

Note that for <see> and <seealso>, they are actually creating links
*within* the index, pointing to other parts of the index.  For these we
can either describe their position by creating an id for them or by
specifying the full index term they link to.  I'm not sure which is
better, but introducing id's seems reasonable.

For example, one might use:

<indexdoc>
  <primary>
     <title>Apple</title>
     <secondary>
       <title>Big</title>
       <tertiary>
          <title linkid="idx-a1">Green</title>
          <title linkid="idx-a3">Blue</title>
          <seealso seeid="sa-1">New York City</seealso>
       </tertiary>
     </secondary>
   </primary>
   <primary>
     <title>Banana</title>
     <secondary>
       <title linkid="idx-a2">Small</title>
     </secondary>
   </primary>
   <primary>
     <title id="sa-1" linkid="idx-a5">New York City</title>
   </primary>
</indexdoc>

or

<indexdoc>
  <primary title="Apple">
    <secondary title="Big">
      <tertiary title="Green" linkid="idx-a1"/>
      <tertiary title="Blue" linkid="idx-a3"/>
      <seealso title="New York City" seeid="sa-1"/>
    </secondary>
  </primary>
  <primary title="Banana">
    <secondary title="Small" linkid="idx-a2"/>
  </primary>
  <primary title="New York City" id="sa-1" linkid="idx-a5">
  </primary>
</indexdoc>

So, here SK generated the id "sa-1" which is used to link to "New York
City" in the index, while "idx-a5" is the id assigned to the index term by
the author for linking from the index into the body of the document.

Does this seem reasonable?

Dan