scrollkeeper-devel — Developer discussion list for ScrollKeeper

[Scrollkeeper-devel] SK mini-roadmap

[Dan M. <da...@ea...>]

Laszlo,

I'm trying to assemble our plans just a bit.  It looks like we have 4
developments in the near future:

A) Move SK to libxml2
B) i18n improvements
C) Support relocatable packages
D) indexing/searching

Am I missing anything we need to do in the short term?

Is this the order we plan to implement things?

What is the status on (A).  Is (A) pretty hard or going to take a lot of
time? 

I don't see much point in making a new stable release for (A) since it
doesn't have user-visible changes.  I'm tempted to make a new release for
each of the other points unless (C) is easy/fast enough to lump in with
(B) without delaying it much.  So, I have:

   Release Number         Changes            Est. Release Date
   --------------         -------            -----------------
    0.3.x / 0.4         (A) and (B)             late May
    0.5.x / 0.6             (C)	                late June
    0.7.x / 0.8             (D)	                mid July

(These dates are very rough of course.)

If we think (A) will take a long time, we may want to do (B) first since
we get important functionality from (B).  However, if Mary needs (A) to do
(D), we may have to do (A) before (B).

Dan      

[Scrollkeeper-devel] Better internationalization support

[Laszlo K. <las...@Su...>]

Hi,

I would like to implement this, but I think we still have some issues to
clarify. I think there are two problems here:

1. Using xml-i18n-tools to translate the category trees.
2. Being able to identify a doc and all of its translations easily.

1. depends on xml-i18n-tools that (I think) creates a message file from
the XML file and merges back all the translations into the original
file. This will be changed in the near future to create separate XML
files for each translation. It seems we will not use it until this is
not done so translation will be done by hand until then.

2. This is the long discussed document ID and this is what I would like
to address. We want an ID that is the same a document and all of its
translations and it allows to find the English doc if the translated one
is not there or fall back to other translations of the doc (like first
to German then to English if the French is not there. I think the main
problem is not how to implement the fallback mechanism, but how to
assign these IDs, how to make the docwriters, translators to request the
ID and how to send it to them. It should be very easy for anybody to
generate an ID like this. Then I think it should be down to the
docwriters, translators to make sure to use the same ID for a doc and
all of its translations. Uidgen was mentioned to be used to generate the
IDs. As this is not available on every platform a web based request
mechanism with a Linux server could be a good approach (like ask the
Linux server for a number). This should probably all be hidden inside a
Scrollkeeper script or binary that just retrieves an ID whenever run. 

Any thoughts?

Laszlo

[Scrollkeeper-devel] Re: ScrollKeeper

[Eric B. <er...@ca...>]

On Fri, Apr 13, 2001 at 02:07:59AM -0700, Bob Stayton wrote:
> I'm looking forward to working with Scrollkeeper
> and the OMF metadata.  I read over the latest SK design
> document and the project is coming together.
>=20
> As you and Cornelius point out, generating the OMF
> data will be a big task.  Content in Docbook will
> probably be the easiest.  If we want to integrate
> doc available in other mime types, we'll have to look at
> ways of extracting metadata from them as well.  There is a
> lot of doc available only as HTML sans OMF that will still
> need to be integrated.  And I doubt it will be XHTML. 8^)
>=20
> For me, having the merged metadata in XML
> suggests a need to build up the XML tools
> on OpenLinux.  I think OpenLinux needs a good
> XML toolkit that could be used for many purposes.
> If we are going to be a development platform,
> we pretty much need one.  Do you know if
> there are plans being developed for such a toolkit?
>=20
> Eric, I saw your mail about putting together an XSL
> kit for Docbook.  I'd be willing help out with that.
> I like using your jw tool.  I'm currently using it
> to generate HTML for all of these:

This new kit will be rather an big change. The old one
was based on jade and the style sheets were written in
DSSSL; the printed output was generated via TeX (hence
the many problems with Asian fonts, for example).
The new kit will be based on Xalan/Xerces and use XSL
for the style sheets; the printed output will be
generated via FOP. Both kits will benefit (at least
I hope so) from an unified inteface in jw.

> Kathleen's Installation Guide   (Docbook SGML 2.2.1 from Frame files)
> Patrice's SysAdmin Guide        (Docbook SGML 2.2.1 from Frame files)
> Meg's Developer Workstation Guide  (Docbook SGML 3.1, hand coded)
> Mark Forry's Open Messaging Server (Docbook XML 4.1 from XMetal)
>=20
> Can you believe that Frame SGML 6 still uses version
> 2.2.1 as their Docbook app?  That DTD came out in 1995!

I wouldn't be *that* surprised.

> No wonder people don't take Frame's SGML efforts seriously.
> Fortunately, all I have to do is change the public id
> in the files to 3.1 and jw works fine.
>=20
> BTW, I found this rather astonishing line in the LDP
> Author Guide Chapter 4 (thanks to an htdig search for XSL
> on the latest Sybil):
>=20
> $JADE -t sgml -i xml -d /home/ldp/SGML/style/xsl/docbook/html/docbook.x=
sl $DOCUMENT.sgml
>=20
> This sure looks to me like jade is processing an SGML file
> with an xsl stylesheet, but that would be a pretty good
> trick, don't you think?   8^)

Hey, if someone can do that, I'm more than happy ;-).

No, seriously, this must be a bug. Jade is a DSSS=B5=B5L engine, not
an XSL engine.

--=20
=C9ric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] Re: ScrollKeeper

[Norman W. <nd...@nw...>]

/ Paul Jones <pj...@me...> was heard to say:
| http://ibiblio.org/osrt/omf/
| open source metadata framework as discussed at the doc summit last summer.

Right. Sorry. Brain cramp.

| On 16 Apr 2001, Norman Walsh wrote:
| 
| +/ Eric Bischoff <er...@ca...> was heard to say:
| +| I also suggested that on the long run the OMF data become
| +| part of the standard DocBook, do you remember, Norm?
| +
| +I'm offline at the moment, so I can't ask Google. What's OMF?

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <nd...@nw...> | If we lived alone in a featureless
http://nwalsh.com/            | desert we should learn to place the
                              | individual grains of sand in a moral or
                              | aesthetic hierarchy.--Michael Frayn

Re: [Scrollkeeper-devel] Re: ScrollKeeper

[Paul J. <pj...@me...>]

http://ibiblio.org/osrt/omf/
open source metadata framework as discussed at the doc summit last summer.

On 16 Apr 2001, Norman Walsh wrote:

+/ Eric Bischoff <er...@ca...> was heard to say:
+| I also suggested that on the long run the OMF data become
+| part of the standard DocBook, do you remember, Norm?
+
+I'm offline at the moment, so I can't ask Google. What's OMF?
+
+                                        Be seeing you,
+                                          norm
+
+

==========================================================================
                             Paul Jones
                    "Gort! Klaatu barada nikto!"
http://www.ibiblio.org/pjones/ at the Site Formerly Known As MetaLab.unc.edu
  pj...@ib...   voice: (919) 962-7600     fax: (919) 962-8071
===========================================================================

[Scrollkeeper-devel] Re: ScrollKeeper

[Norman W. <nd...@nw...>]

/ Eric Bischoff <er...@ca...> was heard to say:
| I also suggested that on the long run the OMF data become
| part of the standard DocBook, do you remember, Norm?

I'm offline at the moment, so I can't ask Google. What's OMF?

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <nd...@nw...> | It is seldom that any liberty is lost
http://nwalsh.com/            | all at once.--David Hume

[Scrollkeeper-devel] Hallo

[Knud H. <no...@ae...>]

Hi Scrollkeepers.

I was with excitement that I read about 
your project in the summary from GUADEC.
The reason is that I have been wanting  to 
start a similar project for gnome.

I have already made my own search engine 
based on similar principles.
Check out http://www.aeiwi.com/

It users title, description and keywords to build a web index.

Right now it contains 700000 links and growing.

Check out the navigation system which is automatic 
generated form the keywords and allows the user to filter
trough the links to they have a small and relevant set of 
documents.

If you like we could work together and provide a similar 
system with scrollkeeper.


Knud

[Scrollkeeper-devel] Scrollkeeper 0.2 design document

[Laszlo K. <las...@su...>]

Hi All,

 We finally managed to create our design document for release 0.2 of
Scrollkeeper. This is a rather weird document as it does not describe
what we will implement, but what we already implemented as Scrollkeeper
0.2 has been released a couple of weeks ago.

Here is the link:

http://scrollkeeper.sourceforge.net/data/scrollkeeper_0.2_specs.txt

Laszlo

[Scrollkeeper-devel] Re: ScrollKeeper

[Eric B. <er...@ca...>]

On Mon, Apr 09, 2001 at 04:50:10PM +0200, Cornelius Schumacher wrote:
> Hi Eric,
>=20
> I had a deep look into ScrollKeeper and now I have some questions to yo=
u.=20
>=20
> There are three things we would have to do to make use of ScrollKeeper:
>=20
> 1) provide OMF meta files for the documentation

I raised this point when the scrollkeeper project started. I said that
this information should be extracted as much as possible from the DocBook=
 file.
This includes the abstract, the keywords, etc. It means that we need a sc=
ript to
extract it.=20

For the OMF data that are not in the DocBook file, it could stand in a se=
parate file
along with the DocBook file, just like we had NIF files at the side of th=
e
LinuxDoc files. Or even better, we could extend the DocBook DTD to accomo=
date
for the new tags, do you agree Frederik?

I also suggested that on the long run the OMF data become
part of the standard DocBook, do you remember, Norm?

> 2) provide a documentation browser, which can read the ScroolKeeper dat=
abase
> 3) provide some support for the existing documentation browsers like=20
> khelpcenter and docview.
>=20
> 3) is currently addressed by doctool.

The new doc browser from Coolo which can render DocBook files directly
might also be a good place to do this.

> For 2) we would have to extend or rewrite khelpcenter, that's feasible

khelpcenter needs some maintenance anyway. For example, with the new po m=
echanism
we will have English section identificators only and we can remove the an=
chors
mechanism then.

> and 1) is ... a lot of work.

Adding the meta information and translating it is a big work, but the ext=
ra software
infrastructure is not.

Yes, since we are already doing a huge translation effort with DocBook
files, moving to XML, etc, we might need to postpone a part of this effor=
t.

But please note that we have been doing much bigger things than this alre=
ady,
and that we can do it progressively.
=20
> The GNOME people apparently already include OMF data in at least a subs=
et of=20
> their packages and they use Nautilus as documentation browser, which su=
pports=20
> reading from the ScrollKeeper database.
>=20
> How do you think about using ScrollKeeper in the context of KDE?

I strongly support it.

> As I've=20
> understood, ScrollKeeper is designed to be independent of a specific de=
sktop=20
> environment, but integrating the required meta information into the KDE=
=20
> documentation is a big task. Do you think we should do that?

Definitly, yes, because it's increased quality for the end user. When and=
 how we
do it is another topic.

--=20
=C9ric Bischoff  -  Documentation and Localization
Caldera (Deutschland) GmbH - Linux for eBusiness
Tel: +49 9131 7192 300 -  Fax: +49 9131 7192 399
http://www.caldera.de/

Re: [Scrollkeeper-devel] Plan for Category Tree for ScrollKeeper >= 0.4

[John F. <jf...@sw...>]

On Fri, Mar 30, 2001 at 01:12:53AM -0800, Dan Mueth wrote:
> 
>   B) mapping between C and non-C locales for categories.  This is probably
>      required in our new po-file technique of i18n.  (Note that now, the C
>      and non-C locales could have completely unrelated entries in theory.)
> 
>      Question: What should we use for our key?  Should it just be the 
>      category in C?  Or should we have a seperate attribute containing
>      some sort of a key?
> 

An attribute with a key would be more complicated to implement, but
would guarantee uniqueness.

>      * This (along with document mappings) allows us to merge
>        the category trees for two locales.  Suppose for example, that
>        there is a document installed only in C.  Using the document 
>        mapping, we could identify that it is not available in locale <xx>
>        and determine which localized category the C doc belongs under.
>        This is a feature we really need in 0.4, and the first feature
>        request I heard from translators.
> 

This would be of great value.

>   C) OMF-file category compatibility between C and non-C locales. So if an
>      OMF file for locale <xx> has a category of "string", then it tries to
>      match the "string" with the localized category tree as well as with
>      the C tree.  This way people don't have to translate a category in
>      the OMF file if they don't want to, and everything will still work.
> 
>      * This is just a practical issue.  Sometimes people forget to
>        translate the category, so this is extra rubustness. Plus, life
>        is easier if we have less stuff to translate, so we may even
>        encourage people to not translate the category for the OMF file
>        if they don't want to.
> 

This would solve a major problem I have had in creating omf files, but
authors writing in other than English should be able to use their own
language for the category name.

> 
> 4) Introduce the proper XML notation for the OMF files for categories,
> instead of using our pipe-delimeted form which is lame.  We should remain
> backward compatible with the pipe form, but strongly discourage it since
> it is inconsistant with the XML concept.
> 

Could you give an xml example of what this would look like?

> 
> 7) (not really category-specific) Look into character encoding issues and
> compatibility with the new (as-yet-unreleased) libxml.
> 

This probably needs to be done right away, doesn't it, so we are ready
for the release of the new libxml?

Cheers,
-- 
John Fleck
jf...@sw... (h)
jf...@ab... (w)
http://www.abqjournal.com/scitech/

Re: [Scrollkeeper-devel] Plan for Category Tree for ScrollKeeper >= 0.4

[Laszlo K. <las...@Su...>]

Dan Mueth wrote:
> 
> Hi,
> 
> I have been thinking a bit about the category tree and how we can should
> move forward.  Here are a few of my thoughts for what we should do between
> now and ScrollKeeper 0.4:
> 
> 1) Avoid changing any existing categories which would break compatibility
> with existing OMF files.  We can happily break this rule if we decide to
> do #3 below, after we implement it :)
> 
> 2) Reasonable i18n.  We should use xml-i18n-tools or some nice way of
> making translators lives easier.  This may include:
> 
>   A) po files to make translation of categories easy
> 
>      * this makes the lives of translators much easier and happier
xml-i18n-tools means one tree that holds the docs for every locale and
all the branch translations. Might be a good idea, but it might
introduce some overhead. If we dont want the browser to filter out the
docs for the current locale then the content tree retrieval scripts will
have to do this. Then we might need to return the XML tree as
represented by libxml in memory rather than a p[ointer to the content
tree file itself. I dont think reading in the big XML tree, filtering
out the current locale, writing that out to a temporary file and
returning that is a good idea because it will be slow. 

> 
>   B) mapping between C and non-C locales for categories.  This is probably
>      required in our new po-file technique of i18n.  (Note that now, the C
>      and non-C locales could have completely unrelated entries in theory.)
> 
>      Question: What should we use for our key?  Should it just be the
>      category in C?  Or should we have a seperate attribute containing
>      some sort of a key?
> 
>      * This (along with document mappings) allows us to merge
>        the category trees for two locales.  Suppose for example, that
>        there is a document installed only in C.  Using the document
>        mapping, we could identify that it is not available in locale <xx>
>        and determine which localized category the C doc belongs under.
>        This is a feature we really need in 0.4, and the first feature
>        request I heard from translators.
Are you suggesting a document mapping and a tree category mapping
database? It is certainly possible to implement it and it would easily
allow us to find the various translations of a doc and add the English
instead of the translated one in any locale. The document ID is needed
to be implemented for this.

> 
>   C) OMF-file category compatibility between C and non-C locales. So if an
>      OMF file for locale <xx> has a category of "string", then it tries to
>      match the "string" with the localized category tree as well as with
>      the C tree.  This way people don't have to translate a category in
>      the OMF file if they don't want to, and everything will still work.
> 
>      * This is just a practical issue.  Sometimes people forget to
>        translate the category, so this is extra rubustness. Plus, life
>        is easier if we have less stuff to translate, so we may even
>        encourage people to not translate the category for the OMF file
>        if they don't want to.
I dont understand everything above. I guess you are suggesting that with
the above mentioned category mapping we can install a translated doc in
the tree into the right locale position even if the OMF category fields
are not translated? Why would one want to match the translated doc with
the C tree?

> 
> 3) Backwards compatibility in the category tree.  Ideally we would have a
> perfect category tree and never change it.  That is not reality.  Really,
> we will get very frequent requests for changes to be made.  These
> potentially break existing OMF files.  However, I think I have a
> reasonable way to change the category system while remaining backwards
> compatible:
> 
> Conceptually we would need a set of instructions for what the obsoleted
> categories are and how they should be mapped. For example, suppose we used
> to have a category "GNOME|Applets|Utility" and we changed the name to
> "GNOME|Applets|Utilities", we would specify (in a seperate XML file
> probably - perhaps scrollkeeper_cl_aliases.xml) that anything that says it
> is in "GNOME|Applets|Utility" should be placed into
> "GNOME|Applets|Utilities".
> 
> The file scrollkeeper_cl_aliases.xml might look like:
> <?xml version="1.0"?>
> <ScrollKeeperContentsAliasList>
>  <omfalias>
>   <oldcategory>
>    <sect>
>     <title>GNOME</title>
>     <sect>
>      <title>Applets</title>
>      <sect>
>       <title>Utility</title>
>      </sect>
>     </sect>
>    </sect>
>   </oldcategory>
>   <oldcategory>
>    <sect>
>     <title>GNOME</title>
>     <sect>
>      <title>Applets</title>
>      <sect>
>       <title>Utility</title>
>      </sect>
>     </sect>
>    </sect>
>   </oldcategory>
>  </omfalias>
> </ScrollKeeperContentsAliasList>
> 
> This is rather bulky to type, but should do the job nicely.  We really
> need to do this in such a way that translators don't have to manually
> translate this.  This can either be done by using the translated terms in
> po files, since they are already translated, or possibly by using the
> correspondence system above.  This will require a bit more thought.
Sounds good, it has to allow more than one old category mapped to the
current new category. 

> 
> 4) Introduce the proper XML notation for the OMF files for categories,
> instead of using our pipe-delimeted form which is lame.  We should remain
> backward compatible with the pipe form, but strongly discourage it since
> it is inconsistant with the XML concept.
yes.

> 
> 5) Start making and publishing official ScrollKeeper category trees.  They
> should be on the web page in an easy-to-read and easy-to-use form, with
> versioning info.
yes.

> 
> 6) Get the OMF specification and DTD updated to reflect the category
> child to SUBJECT
yes.

> 
> 7) (not really category-specific) Look into character encoding issues and
> compatibility with the new (as-yet-unreleased) libxml.
yes. DV should be pulled into this.

> 
> This is a lot of stuff, so some of it may have to wait for ScrollKeeper
> 0.6, depending on how long it takes to get this done.
> 
> Thoughts?

As I said to Dan before, priorities in the Sun Gnome team were
reshuffled and I will be able to spend about 30% of my time on
Scrollkeeper (until know this was 100%). So we will need to take design
related decisions much faster in order to get things to the stage when
they can be implemented considering that only myself and another Sun
engineer are implementing features in Scrollkeeper right now.

Laszlo

[Scrollkeeper-devel] Plan for Category Tree for ScrollKeeper >= 0.4

[Dan M. <da...@ea...>]

Hi,

I have been thinking a bit about the category tree and how we can should
move forward.  Here are a few of my thoughts for what we should do between
now and ScrollKeeper 0.4:

1) Avoid changing any existing categories which would break compatibility
with existing OMF files.  We can happily break this rule if we decide to
do #3 below, after we implement it :)

2) Reasonable i18n.  We should use xml-i18n-tools or some nice way of
making translators lives easier.  This may include:

  A) po files to make translation of categories easy

     * this makes the lives of translators much easier and happier

  B) mapping between C and non-C locales for categories.  This is probably
     required in our new po-file technique of i18n.  (Note that now, the C
     and non-C locales could have completely unrelated entries in theory.)

     Question: What should we use for our key?  Should it just be the 
     category in C?  Or should we have a seperate attribute containing
     some sort of a key?

     * This (along with document mappings) allows us to merge
       the category trees for two locales.  Suppose for example, that
       there is a document installed only in C.  Using the document 
       mapping, we could identify that it is not available in locale <xx>
       and determine which localized category the C doc belongs under.
       This is a feature we really need in 0.4, and the first feature
       request I heard from translators.

  C) OMF-file category compatibility between C and non-C locales. So if an
     OMF file for locale <xx> has a category of "string", then it tries to
     match the "string" with the localized category tree as well as with
     the C tree.  This way people don't have to translate a category in
     the OMF file if they don't want to, and everything will still work.

     * This is just a practical issue.  Sometimes people forget to
       translate the category, so this is extra rubustness. Plus, life
       is easier if we have less stuff to translate, so we may even
       encourage people to not translate the category for the OMF file
       if they don't want to.

3) Backwards compatibility in the category tree.  Ideally we would have a
perfect category tree and never change it.  That is not reality.  Really,
we will get very frequent requests for changes to be made.  These
potentially break existing OMF files.  However, I think I have a
reasonable way to change the category system while remaining backwards
compatible:

Conceptually we would need a set of instructions for what the obsoleted
categories are and how they should be mapped. For example, suppose we used
to have a category "GNOME|Applets|Utility" and we changed the name to
"GNOME|Applets|Utilities", we would specify (in a seperate XML file
probably - perhaps scrollkeeper_cl_aliases.xml) that anything that says it
is in "GNOME|Applets|Utility" should be placed into
"GNOME|Applets|Utilities".

The file scrollkeeper_cl_aliases.xml might look like:
<?xml version="1.0"?>
<ScrollKeeperContentsAliasList>
 <omfalias>
  <oldcategory>
   <sect>
    <title>GNOME</title>
    <sect>
     <title>Applets</title>
     <sect>
      <title>Utility</title>
     </sect>
    </sect>
   </sect>
  </oldcategory>
  <oldcategory>
   <sect>
    <title>GNOME</title>
    <sect>
     <title>Applets</title>
     <sect>
      <title>Utility</title>
     </sect>
    </sect>
   </sect>
  </oldcategory>
 </omfalias>
</ScrollKeeperContentsAliasList>

This is rather bulky to type, but should do the job nicely.  We really
need to do this in such a way that translators don't have to manually
translate this.  This can either be done by using the translated terms in
po files, since they are already translated, or possibly by using the
correspondence system above.  This will require a bit more thought.

4) Introduce the proper XML notation for the OMF files for categories,
instead of using our pipe-delimeted form which is lame.  We should remain
backward compatible with the pipe form, but strongly discourage it since
it is inconsistant with the XML concept.

5) Start making and publishing official ScrollKeeper category trees.  They
should be on the web page in an easy-to-read and easy-to-use form, with
versioning info.

6) Get the OMF specification and DTD updated to reflect the category
child to SUBJECT

7) (not really category-specific) Look into character encoding issues and
compatibility with the new (as-yet-unreleased) libxml.

This is a lot of stuff, so some of it may have to wait for ScrollKeeper
0.6, depending on how long it takes to get this done.  

Thoughts?

Dan

[Scrollkeeper-devel] question re: marking index terms

[Mary D. <Mar...@Su...>]

hi 

I have a question on the type of index markers that scrollkeeper can support.
Scrollkeeper needs the index marker to use an id to facilitate the "jump" from 
the index to the correct location in the document.

DocBook has two kinds of index markers: 
- singular 
and 
- multiple

1. The singular index marker is inserted where the subject occurs in the text, 
eg.
<para>
The tiger<indexterm>
<primary>Big Cats</primary>
<secondary>Tigers</secondary></indexterm>
is a very large cat indeed.
</para>

This method does not use an id. 

2. There are two ways of using the multiple index marker:
   a) put index terms at start and end of a range of text, eg
      
<para>
The tiger<indexterm id="tiger-desc" class="startofrange">
<primary>Big Cats</primary>
<secondary>Tigers</secondary></indexterm>
is a very large cat indeed&hellip;
</para>
&vellip;
<para>
So much for tigers<indexterm startref="tiger-desc" class="endofrange">. Let's 
talk about
leopards.  
</para>

This method uses an id, and the indexterm markers are within context in the 
document.

   b) the other way to mark a range of text is to use the Zone attribute.
      This method specifies the content of an element (eg chapter, section) 
      as the complete range. The index term therefore points to the id of
      the element.
This method uses an id, but the index marks do not have to occur near the text
being indexed.
- this would not be of use to scrollkeeper.
However, if the index marker was inserted at start of element it indexed then it 
could be used?


Ok, those are my assumptions, I would be grateful if people could 
confirm/correct them so that I can ascertain how index markers
should be used in scrollkeeper-compatible documents.

TIA
cheers
Mary




~ I speak for myself, not for my employer ~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Mary Dwyer
Desktop Applications & Middleware Grp
Sun Microsystems Ireland
Tel: +353-1-8199222 (xt 19222)
Fax: +353-1-8199078
email: mar...@ir...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@Su...>]

> Here are a few of my thoughts on what sort of functionality a user may
> want.
> 
> 1) "Basic Document Index"
> 
> For any document, a user may want an index.  Here I mean "index" as what
> one normally finds at the back of a book.  It is a list of terms, often
> with secondary or even tertiary headings and "see also" items.  These
> terms link to concepts and/or words in the body of the document.  The
> purpose is to allow a user who is looking for information on a particular
> concept to quickly find the part(s) of the document which discuss this
> concept.
> 
> DocBook provides tags to markup terms to appear in an index.  When the
> document is processed, such as when one converts the SGML to HTML or PS,
> it generates the index from the many marked up terms throughout the
> document.  For HTML, a link is provided from the index to the correct part
> of the document.  For PS, I believe the page number is listed.
> 
> This is a very useful level of indexing which is already possible without
> ScrollKeeper.  The only requirements are that an experienced indexer
> manually add the indexing markup and that the document processor knows how
> to generate the index.  (Note this requires a fair amount of expertise and
> time on the part of the indexer, but I believe the technology is complete
> for DocBook/SGML documents rendered into the standard target formats.)
> 
> 2) "Help System Aware Document Index"
> 
> To increase usability, the help browser is aware of the document index
> from #1, rather than merely displaying the content of the index.  This
> allows the user to do things like:
> 
> a) browse the index in one window or portion of the help browser while
> viewing the document in another
> 
> b) more quickly navigate and search the index using an index browser built
> into the help system
> 
> c) search multiple documents to identify which one discusses a certain
> topic, or even create compound indexes.
> 
> d) (many other things which are possible, but which probably should be
> done in ScrollKeeper instead of the help browser.  I discuss some of them
> below.)
> 
> I am mainly interested in (a) and (b) above, since I think (c) is probably
> something which should be in (d).
> 
> 3) "System-wide Index"
> 
> This is an index of all documents on the system, obtained my merging all
> of the indexes to all of the documents (of a given locale and excluding
> duplicates of a given document) on the system.
> 
> A system-wide index may be valuable for searching for more obscure things,
> like "beowulf".  However, putting in "cd-rom" or "printer" will likely
> return so many items it may be completely useless to the user.
> 
> 4) "Group Indexes"
> 
> This is an index of a certain set of documents which may not span the
> entire set of documents.  For example, it may be an index for all GNOME
> documents, or all KDE documents, or all documents written by John.  I
> believe this will be the most valuable index type.
> 
> In fact, for GNOME I see the need for at least two different groups.  One
> for the core desktop, and another for all GNOME documents.  Similar to the
> example above, it would be nice to enter "printer" and get only a few hits
> about configuring and using printers, not the "how to print from
> application foo" for every application (or every GNOME application) on
> your computer.  So generally I would expect a help browser to request
> indexes from ScrollKeeper for multiple groups of documents.

Sounds good.

> 
> A few thoughts on technical requirements:
> 
> 1) Index Targets
> 
> We need to be able to parse the DocBook/SGML documents and extract
> their indexing terms and anchors.  Identifying the indexing terms should
> be straightforward.  The other half is predicting the anchors in the
> document.  This is another example of where we require our document
> processors to behave in a uniform way.  For example, if we have two
> DocBook to HTML converters, we need them both to label their anchors the
> same way.
> 
> Also, I think we may soon have a good way to render DocBook/XML documents
> directly.  I'm not sure how we would specify the targets in this case.
> (Could anybody provide some information on how this would work?)
> 
> We will need to resolve this issue to achieve any of the 4 functional
> points listed above except #1.

I guess the way how the browser is able or not to jump to the index ID
provided is up to the browser, not Scrollkeeper. We just have to be sure
that we push the usage of those index terms from SGML that facilitate
the implementation of the jump to the right index content.

> 
> 2) General Database and API
> 
> We should make the index database as general as possible.  Ideally, for
> each occurance of each indexing term, we associate all of the OMF metadata
> of the document.  This allows the help browser to make arbitrary queries,
> from requesting an index for a single document, to an index of the whole
> system, to an index of an arbitrary list of documents, to an index with
> certain metadata (eg. documents with a <subject> of "Core GNOME
> Documents").
> 
> This would mean we want to select a free database which ScrollKeeper will
> use to manage all of the indexing data.  The exported ScrollKeeper API for
> returning various indexes would probably be a rather thin wrapper around
> the database.  It also means that ScrollKeeper will leave the innocent
> world of scripts and command-line programs which point to XML files and
> at least the indexing portion will need to be a proper library which
> browsers link to.
> 
> This approach is probably a fair amount more work than using XML files as
> we have in the past.  It will also require more up-front investment in
> terms of design and expertise.  However, I think we need the speed and
> flexibility of a real database in order to provide some of the most
> valuable functionality: The ability to produce indexes on specific groups
> of documents, specified by their metadata.

I think the database is not feasible in the near future. I think the two
most obvious queries are searching for a term in one doc's index and
searching in a group of docs' indexes where the docs are grouped
according to their metadata subject field (the same that is used to add
them to the right place in the category tree). I am sure people can come
up with loads of useful queries, but I think these will be far the most
common. As we have the docs in the category tree ordered by subject,
grouping according to subject is easy. Merging a number of indexes into
a temporary one will not be very difficult. And this is all we need.

The problem with the free database approach is that we dont know which
one is available on the most OS-es, we certainly know that none right
now on Solaris. While this might be sorted in the future, it is
certainly not very close. The approach I described above can be
implemented XML based in 2-3 months and it will work with a not very
large amount of docs. A database approach would take much longer. I dont
exclude the possibility of a parallel XML/database approach or moving
totally to a database though.

Laszlo

Re: [Scrollkeeper-devel] BOF about Scrollkeeper at GUADEC?

[Dan M. <da...@ea...>]

On Fri, 23 Mar 2001, Laszlo Kovacs wrote:

> Hi,
> 
> This might be interesting for people who will attend GUADEC in Denmark
> at the beginning of April.
> 
> Would people who attend the conference be interested in an informal talk
> about Scrollkeeper? Where it is now, what do we expect from it, in what
> direction we want to develop it?  I am thinking of a developers talk
> rather than some kind of presentation. However a short introductory
> presentation might be helpful.
> 
> We might be too late with it to get in the program, but I am sure an
> informal talk can be quite easily organized.
> 
> Laszlo


I think this is a great idea, and I will certainly attend.
 
Starting with a presentation about what ScrollKeeper does and what
the plans for its future are sounds like a good idea, because I think a
lot of people don't know a lot about it.  Plus, I think the most exciting
part is the various features which will be possible in the future, so it
would be nice to hear about these some.

The other thing which may be of interest is motivation.  It would be nice
to start with a little discussion about the needs of the user and of the
help browser, and how ScrollKeeper can be used to meet these needs in a
standardized way that various help browser and documentation projects can
share and use as an interoperability layer.

Just to make even more work out of this, another idea is to write up a
short paper which follows the same themes as the talk/BOF for the benefit
of those who don't attend.

Dan

Re: [Scrollkeeper-devel] BOF about Scrollkeeper at GUADEC?

[Laszlo K. <las...@su...>]

Hi,

Suggestions about how to do it, what to talk about are welcome.

Thanks,

Laszlo

Re: [Scrollkeeper-devel] BOF about Scrollkeeper at GUADEC?

[Ismael O. <ol...@hi...>]

Laszlo Kovacs escribió:
> 
> Hi,
> 
> This might be interesting for people who will attend GUADEC in Denmark
> at the beginning of April.
> 
> Would people who attend the conference be interested in an informal talk
> about Scrollkeeper? Where it is now, what do we expect from it, in what
> direction we want to develop it?  I am thinking of a developers talk
> rather than some kind of presentation. However a short introductory
> presentation might be helpful.
> 
> We might be too late with it to get in the program, but I am sure an
> informal talk can be quite easily organized.
> 

Count with me.

-- 

        A.Ismael Olea González

        mailto:ol...@hi...  http://www.hispalinux.es/~olea
        2:345/10...@fi...
                
        El mundo debe empezar a tener miedo a un planeta OLEA

[Scrollkeeper-devel] BOF about Scrollkeeper at GUADEC?

[Laszlo K. <las...@su...>]

Hi,

This might be interesting for people who will attend GUADEC in Denmark
at the beginning of April.

Would people who attend the conference be interested in an informal talk
about Scrollkeeper? Where it is now, what do we expect from it, in what
direction we want to develop it?  I am thinking of a developers talk
rather than some kind of presentation. However a short introductory
presentation might be helpful.

We might be too late with it to get in the program, but I am sure an
informal talk can be quite easily organized.

Laszlo

[Scrollkeeper-devel] Re: Index support in Scrollkeeper

[Mary D. <Mar...@Su...>]

hi 

Quick question.
According to the docbook guide, the indexterm element can be used to mark either 
a single point in a document, or a range. If a range is used then either the 
startref or zone attribute is used.
Has there been any discussion on how indexterm should be used by opensource doc 
writers? Are there many doc writers out there currently using the indexterm 
element in their docs? Are they using all three of the methods described above, 
or any other methods
Any pointers would be appreciated

cheers
Mary Dwyer



------------- End Forwarded Message -------------


~ I speak for myself, not for my employer ~
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Mary Dwyer
Desktop Applications & Middleware Grp
Sun Microsystems Ireland
Tel: +353-1-8199222 (xt 19222)
Fax: +353-1-8199078
email: mar...@ir...
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

[Scrollkeeper-devel] Re: [Mailer-Daemon@lists.sourceforge.net: libxml version?

[Karl E. <ke...@gm...>]

st...@st... (Steven Fines) writes:


> What version of libxml should I be linking against when I build
> scrollkeeper-0.1.4? Currently I have libxml symlinked to
> libxml2.so.2.3.4.

Don't do this ("Doctor, it hurts...").  Please, install libxml.  These
days it's possible to install both versions at the same time.

-- 
work : ke...@su...                          |                   ,__o
     : http://www.suse.de/~ke/             |                 _-\_<,
home : ke...@gm...                     |                (*)/'(*)

Re: [Scrollkeeper-devel] [Mailer-Daemon@lists.sourceforge.net: libxml version?

[Gregory L. <gle...@cu...>]

On 20 Mar 2001 22:48:03 -0600, Steven Fines wrote:
> All, 
> What version of libxml should I be linking against when I build scrollkeeper-0.1.4? Currently I have libxml symlinked to libxml2.so.2.3.4. I ask because when I attempt to compile, i get several errors that look like:
> 
> preinstall.c: In function `update_doc_url_in_omf_file':
> preinstall.c:46: structure has no member named `childs'
> preinstall.c: In function `main':
> preinstall.c:81: structure has no member named `root'
> 
> I checked the header file <tree.h> and noticed that none of the structures defined in that library, which is where the xmlNode is defined have a childs member. I went into the code and changed all references from childs to children. That seemed like a fairly safe assumption for me to make. The root member was another problem entirely. I finally settled on using the parent member as the root. I don't think that was correct, but that's not at issue yet. I got the code in the cl/src directory to build and then ran into trouble later on in the process with this error:
> 
> make[3]: Entering directory `/home/steve/eazel_inst/step4/scrollkeeper-0.1.4-fixed/toc/src'
> gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/include/libxml -I/usr/include    -g -O2 -c toc.c
> toc.c: In function `toc_internalSubset':
> toc.c:309: too few arguments to function `xmlDetectCharEncoding'
> make[3]: *** [toc.o] Error 1
> 
> So I'm stumped and left to assume that I have a the incorrect library version. Since 2.3.4 is all that apt-get will give me, I was pondering wether or not it is the correct one and if I need to rustle up the libxml sources and compile them.

Currently, Scrollkeeper is being developed against libxml 1, and not
libxml 2.  I don't know what the plans are, but I don't expect a move to
libxml until mid-summer, at the earliest (December at the latest).

    Greg

[Scrollkeeper-devel] [Mailer-Daemon@lists.sourceforge.net: libxml version?

[<st...@st...>]

All, 
What version of libxml should I be linking against when I build scrollkeeper-0.1.4? Currently I have libxml symlinked to libxml2.so.2.3.4. I ask because when I attempt to compile, i get several errors that look like:

preinstall.c: In function `update_doc_url_in_omf_file':
preinstall.c:46: structure has no member named `childs'
preinstall.c: In function `main':
preinstall.c:81: structure has no member named `root'

I checked the header file <tree.h> and noticed that none of the structures defined in that library, which is where the xmlNode is defined have a childs member. I went into the code and changed all references from childs to children. That seemed like a fairly safe assumption for me to make. The root member was another problem entirely. I finally settled on using the parent member as the root. I don't think that was correct, but that's not at issue yet. I got the code in the cl/src directory to build and then ran into trouble later on in the process with this error:

make[3]: Entering directory `/home/steve/eazel_inst/step4/scrollkeeper-0.1.4-fixed/toc/src'
gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I/usr/include/libxml -I/usr/include    -g -O2 -c toc.c
toc.c: In function `toc_internalSubset':
toc.c:309: too few arguments to function `xmlDetectCharEncoding'
make[3]: *** [toc.o] Error 1

So I'm stumped and left to assume that I have a the incorrect library version. Since 2.3.4 is all that apt-get will give me, I was pondering wether or not it is the correct one and if I need to rustle up the libxml sources and compile them.

Thanks, 
Steve Fines

-- 

----- End forwarded message -----

-- 

Re: [Scrollkeeper-devel] Index support

[Dan M. <da...@ea...>]

Before we get too far into technical details of how to implement indexing,
I'd like to discuss the general ideas for what we want to achieve.  Here I
think we should start from the perspective of the end-user sitting in
front of a help browser.  The help browser I am thinking of is the GNOME
help browser or perhaps the KDE help browser.  However, what we design
must be able to work just as well for a general Linux help browser, *nix
help browser, or web-based help browser for project Foo or organization
Bar.

Here are a few of my thoughts on what sort of functionality a user may
want.

1) "Basic Document Index" 

For any document, a user may want an index.  Here I mean "index" as what
one normally finds at the back of a book.  It is a list of terms, often
with secondary or even tertiary headings and "see also" items.  These
terms link to concepts and/or words in the body of the document.  The
purpose is to allow a user who is looking for information on a particular
concept to quickly find the part(s) of the document which discuss this
concept.

DocBook provides tags to markup terms to appear in an index.  When the
document is processed, such as when one converts the SGML to HTML or PS,
it generates the index from the many marked up terms throughout the
document.  For HTML, a link is provided from the index to the correct part
of the document.  For PS, I believe the page number is listed.

This is a very useful level of indexing which is already possible without
ScrollKeeper.  The only requirements are that an experienced indexer
manually add the indexing markup and that the document processor knows how
to generate the index.  (Note this requires a fair amount of expertise and
time on the part of the indexer, but I believe the technology is complete
for DocBook/SGML documents rendered into the standard target formats.)

2) "Help System Aware Document Index"

To increase usability, the help browser is aware of the document index
from #1, rather than merely displaying the content of the index.  This
allows the user to do things like:

a) browse the index in one window or portion of the help browser while
viewing the document in another

b) more quickly navigate and search the index using an index browser built
into the help system

c) search multiple documents to identify which one discusses a certain
topic, or even create compound indexes.

d) (many other things which are possible, but which probably should be
done in ScrollKeeper instead of the help browser.  I discuss some of them 
below.)

I am mainly interested in (a) and (b) above, since I think (c) is probably
something which should be in (d).

3) "System-wide Index"

This is an index of all documents on the system, obtained my merging all
of the indexes to all of the documents (of a given locale and excluding
duplicates of a given document) on the system.

A system-wide index may be valuable for searching for more obscure things,
like "beowulf".  However, putting in "cd-rom" or "printer" will likely
return so many items it may be completely useless to the user.

4) "Group Indexes"

This is an index of a certain set of documents which may not span the
entire set of documents.  For example, it may be an index for all GNOME
documents, or all KDE documents, or all documents written by John.  I
believe this will be the most valuable index type.

In fact, for GNOME I see the need for at least two different groups.  One
for the core desktop, and another for all GNOME documents.  Similar to the
example above, it would be nice to enter "printer" and get only a few hits
about configuring and using printers, not the "how to print from
application foo" for every application (or every GNOME application) on
your computer.  So generally I would expect a help browser to request
indexes from ScrollKeeper for multiple groups of documents.

--

A few thoughts on technical requirements:

1) Index Targets

We need to be able to parse the DocBook/SGML documents and extract
their indexing terms and anchors.  Identifying the indexing terms should
be straightforward.  The other half is predicting the anchors in the
document.  This is another example of where we require our document
processors to behave in a uniform way.  For example, if we have two
DocBook to HTML converters, we need them both to label their anchors the
same way.  

Also, I think we may soon have a good way to render DocBook/XML documents
directly.  I'm not sure how we would specify the targets in this case.  
(Could anybody provide some information on how this would work?)

We will need to resolve this issue to achieve any of the 4 functional
points listed above except #1.

2) General Database and API

We should make the index database as general as possible.  Ideally, for
each occurance of each indexing term, we associate all of the OMF metadata
of the document.  This allows the help browser to make arbitrary queries,
from requesting an index for a single document, to an index of the whole
system, to an index of an arbitrary list of documents, to an index with
certain metadata (eg. documents with a <subject> of "Core GNOME
Documents").

This would mean we want to select a free database which ScrollKeeper will
use to manage all of the indexing data.  The exported ScrollKeeper API for
returning various indexes would probably be a rather thin wrapper around
the database.  It also means that ScrollKeeper will leave the innocent
world of scripts and command-line programs which point to XML files and
at least the indexing portion will need to be a proper library which
browsers link to.

This approach is probably a fair amount more work than using XML files as
we have in the past.  It will also require more up-front investment in
terms of design and expertise.  However, I think we need the speed and
flexibility of a real database in order to provide some of the most
valuable functionality: The ability to produce indexes on specific groups
of documents, specified by their metadata.

Dan

Re: [Scrollkeeper-devel] Index support

[Laszlo K. <las...@su...>]

> My thinking here was that SK would put the index data into an xml data
> structure that was browser-independent, then people's browsers would
> use that data to generate a displayable version of the
> document-specific index, showing up as an appendix to the document. I
> have some ideas about how we might do that for
> GNOME, but I assume others might have other ideas for other platforms,
> and my thinking was that the SK data structure would be a
> platform-neutral xml structure we all could draw on.
This is precisely how it should work.
 
> As an alternative, I suppose, we could have SK just generate a DocBook
> index document for the doc at install time and stick it in the same
> directory as the doc.
I think we need both. We need index associated with every doc and we
also need joint index of several docs or all the docs from the system.
The basis should obviously be separate index for each doc and then joint
indexes can be built on that easily.
 
> So you've have the large systemwide xml data structure containing all
> index data, and a separate index.sgml (or index.xml) that gets stuck
> in the install directory with the doc. Would that work?
Something like this will work definitely and this is how I think it
should be done.

Laszlo

Re: [Scrollkeeper-devel] omf question

[David M. <da...@lu...>]

On Fri, Mar 16, 2001 at 11:19:08AM -0500, Paul Jones wrote:
> let me take a crack at answering your questions and other omf folks please
> correct me.
> 
> On Thu, 15 Mar 2001, David Merrill wrote:
> 
> +    # You don't have version information in your example.
> +    # I dont' know if ours is right yet, either. This is a pre-alpha
> +    # (non) release.
> +    <versionGroup created="20010315">
> +      <version created="20010315">
> +        <id created="20010315">
> officially you would not be required to enter a version number as it is
> optional information see http://www.ibiblio.org/osrt/omf/omf_elements

Even if it is optional according to omf, I don't think we should
consider it optional for our purposes. It is very valuable information
and scrollkeeper will definitely need it.

> +          # I suspect this is wrong. Too much info. But it could exist
> +	  # in the original docbook, too. Haven't checked that yet.
> +	  # This is why I want to get started doing omf early. There
> +	  # will be wrinkles to work out. We may have to update all
> +	  # our documents' source.
> +          0.6c15 Feb 2001konst
> +        </id>
> +        <date created="20010315">
> +          20010215
> +        </date>
> +      </version>
> +    </versionGroup>
> +
> you are right we don't support a sub-label of category. how would you
> define that?

I have *no* idea. You can see that sk has used a pipe char to separate
them. This seems workable but it would be better to do:

<category>
  <major>System<major>
  <minor>Other</minor>
</category>

or something similar.

> +    # Note that we do not have categories. We need your advice.
> +    # Have you established a system of categorization? I see the
> +    # <subject><category>System|Other</category></subject>, but I am
> +    # wondering what the available values are. Can there be multiple
> +    # categories?
> not a bad idea

I think it's essential.

> +    # We are pulling keywords from the title of the document.
> +    # We will have to come up with a better approach.
> +    <keywords created="20010315">assembly</keywords>
> +    <keywords created="20010315">howto</keywords>
> +    <keywords created="20010315">linux</keywords>
> +
> +    <description created="20010315">
> +       This is the Linux Assembly HOWTO, version &version;. This
> +       document describes how to program in assembly language using
> +       free programming tools, focusing on development for or from
> +       the Linux Operating System, mostly on IA-32 (i386) platform.
> +       Included material may or may not be applicable to other
> +       hardware and/or software platforms.
> +    </description>
> +
> +    # Are you working from the list established in the latest omf
> +    # specification? If so, good. I also just asked for a new value,
> +    # "Quick Reference" since the LDP is going to start working on a
> +    # series of them. Someday.
> would this be a new TYPE? again see
> http://www.ibiblio.org/osrt/omf/omf_elements

Yes, it would. Also, I want to do away with "mini-howto" for various
reasons I've expounded upon many times. Shall I go through them again,
or are you all sick of it? :-)

-- 
Dr. David C. Merrill                     http://www.lupercalia.net
Linux Documentation Project                   da...@lu...
Collection Editor & Coordinator            http://www.linuxdoc.org
                                       Finger me for my public key

All parts should go together without forcing.  You must remember that the parts
you are reassembling were disassembled by you.  Therefore, if you can't get
them together again, there must be a reason.  By all means, do not use a hammer.
		-- IBM maintenance manual, 1925