scrollkeeper-devel — Developer discussion list for ScrollKeeper

Re: [Scrollkeeper-devel] Simple URI's

[Dan M. <d-...@uc...>]

On Mon, 4 Dec 2000, Nik Clayton wrote:

> On Mon, Dec 04, 2000 at 10:10:25AM -0600, Dan Mueth wrote:
> > On Mon, 4 Dec 2000, Laszlo Kovacs wrote:
> > > > Let the users use bookmarks.  On a network with multiple architectures,
> > > > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> > 
> > Bookmarks won't help here.  Bookmarks do not provide a mechanism for one
> > document to cross-reference another, 
> 
> Nor should they.  Document cross references are going to be a pain in
> the backside, which ever way you cut it.  I don't think is really within
> sk's purview.
>
> [ For example, suppose I have a document that references modem-HOWTO in
>   a link.  What happens to the link if modem-HOWTO hasn't been
>   installed?  Is it active (and generates an error if it's clicked), or
>   is it inactive, or what? ]

The name scheme will make things nice enough that people can
cross-reference and that users can type in simple-to-remember URI's.  It  
will not be a perfect system.  I could always create a document which
tries to cross-reference obscure documents and thus will have broken
links.  I don't think the possibility of broken links should scare us
away from a very useful and powerful tool though.  We can always work
around this in various ways.  A smart help browser could check links
before displaying them (ie. throw out broken links).  Or, a help browser
connected to the Net would not generally have broken links if we have a   
large document database (MetaLab).  

At least in the short term, we could just recommend that people link to
documents that they can confidently say will be there.  For example any
GNOME application can link to docs shipped with gnome-core or gnome-libs
or any other document shipped in the given package.  Even if we discourage
cross-referencing documents that are not provided in the package
dependencies we would still have a very useful tool.  I know that with
GNOME, 99% of the links we want to make between documents are guaranteed
to be functional due to package dependencies.  If we have broken links,
then it is our fault, just like if we have broken images, broken
translations, or broken applications.  If we *really* need to make a link
from foo to bar, then we can always list it in the dependencies.

> > nor do they help a user find a document they've never seen before.  
> 
> That's what the index, table of contents, and free text search are for.

Yes.  But people still type in URL's even though they have search engines,
web portals, bookmarks, etc.  It is just another way to find a document
which in many cases is a lot faster and easier than the other
methods.  No, people won't die without it. But yes, it would be a very
useful tool.

Dan

Re: [Scrollkeeper-devel] Simple URI's

[Dan M. <d-...@uc...>]

On Mon, 4 Dec 2000, Ali Abdin wrote:

> * Ali Abdin (ali...@au...) wrote at 22:26 on 04/12/00:
> > * Laszlo Kovacs (las...@su...) wrote at 16:36 on 04/12/00:
> > > > Let the users use bookmarks.  On a network with multiple architectures,
> > > > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> > > > 
> > > > If an individual user wants to create their own aliases for
> > > > documentation, they should probably be able to do so.  The user can then
> > > > have the burden of deciding which short name maps to which document, and
> > > > is in a much better position to know that they want "modem-HOWTO" to
> > > > point to the Solaris document.
> > > > 
> > > 
> > > Or the sysadmin decides where these URI's point to depending on machine
> > > type.
> > 
> > The "short name" should be settable by the sysadmin (for Desktop users, they
> > are the sysadmins). Each doc should have by default its own shortname.
> 
> Oh, doh - if they change the short name, there goes all your cross-references.
> 
> The proper solution is to cross-reference by UUID, but that won't work...no
> authors would do it then...

You don't typically want the completely unique identification (UUID is for
{document, language, version, format}, but the document identifier (NUID
is what I called it) {document but not language, version, or format}.  
This would be mapped onto names to make things "nice" like IP addresses
are mapped onto names.  Of course we won't use a DNS server - we'd put the
name into the metadata and use a database on the web for reference use
only.  It would still be a "first come, first serve" basis for giving out
names (like domain names), although inappropriate use would be frowned
upon of course.

Dan

Re: [Scrollkeeper-devel] questions, questions

[Ali A. <ali...@au...>]

* Gregory Leblanc (gle...@cu...) wrote at 22:28 on 04/12/00:
> > * Bob Stayton (bo...@sc...) wrote at 16:38 on 04/12/00:
> > >
> > > I think the performance of XSL processing will improve
> > > within six to twelve months.  If I were setting up a server
> > > today, though, I would preprocess all my XML docs into HTML
> > > files to be served.  Cron jobs or time stamps would have to
> > > be used to keep the HTML current. This would be similar to
> > > man pages being nroffed into cat files to improve
> > > performance when Unix machines used to be slower.
> > 
> > Yeah, I asked Daniel Veillard about this. Note: Cocoon is in Java as far as I
> > know and we all know about Java performance on Linux (I'm hoping a C
> > implementation would show up so we can compare/benchmark it).
> > 
> > Daniel recommended for me to parse XML using DOM (until XSLT performance
> > improves).
> 
> 
> How does this get around the issues of XSLT being slow?  I'm probably
> being obtuse, but I just don't see the connection.

Well, I was thinking that gnome-db2html2 should use XSL Transformations once
libxml gets the support for it. But Daniel warned me that A) it could take a
while for support and B) is computationally complex (so may not be suited for
run-time processing).

So instead I should use DOM to parse the doucment. (right now we use SAX
libxml parser).
 
> > I wonder how the performance would be if soembody used a "stripped down"
> > version of Norm Walsh's XSL stylesheet (I assume he supports a huge subset of
> > tags (all of DocBook?) while the GNOME project only uses ~120 tags).
> 
> 
> I'd much rather have the full set of DocBook tags available, as the
> overhead of changing the stylesheets when Joe Author decides to use a
> different tag is pretty high.  Does the manner in which XSL stylesheets
> are written affect the speed of processing as much as it does for
> DSSSL?  (One of the GNOME hackers had a REALLY awful DSSSL sheet the
> other day, it took many minutes to run on an 18K file).

Well, since an XSL basically involes some "programming" - it does depend on
how you write the stylesheet. But I think also if you have an XSL with 100
tags and an XSL with 500 tags, the one with 100 tags would be faster (even if
its just by 0.01%)

>     Greg
> 
> 
> P.S.  Hmm, we're probably drifting from the topics that should be on
> sk-devel now...

Yes :) we have diverged quite a bit

Re: [Scrollkeeper-devel] Simple URI's

[Ali A. <ali...@au...>]

* Ali Abdin (ali...@au...) wrote at 22:26 on 04/12/00:
> * Laszlo Kovacs (las...@su...) wrote at 16:36 on 04/12/00:
> > > Let the users use bookmarks.  On a network with multiple architectures,
> > > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> > > 
> > > If an individual user wants to create their own aliases for
> > > documentation, they should probably be able to do so.  The user can then
> > > have the burden of deciding which short name maps to which document, and
> > > is in a much better position to know that they want "modem-HOWTO" to
> > > point to the Solaris document.
> > > 
> > 
> > Or the sysadmin decides where these URI's point to depending on machine
> > type.
> 
> The "short name" should be settable by the sysadmin (for Desktop users, they
> are the sysadmins). Each doc should have by default its own shortname.

Oh, doh - if they change the short name, there goes all your cross-references.

The proper solution is to cross-reference by UUID, but that won't work...no
authors would do it then...

Regards,
Ali

Re: [Scrollkeeper-devel] Simple URI's

[Ali A. <ALI...@au...>]

On Mon, 4 Dec 2000, Nik Clayton wrote:

> On Mon, Dec 04, 2000 at 10:10:25AM -0600, Dan Mueth wrote:
> > On Mon, 4 Dec 2000, Laszlo Kovacs wrote:
> > > > Let the users use bookmarks.  On a network with multiple architectures,
> > > > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> > 
> > Bookmarks won't help here.  Bookmarks do not provide a mechanism for one
> > document to cross-reference another, 
> 
> Nor should they.  Document cross references are going to be a pain in
> the backside, which ever way you cut it.  I don't think is really within
> sk's purview.
> 
> [ For example, suppose I have a document that references modem-HOWTO in
>   a link.  What happens to the link if modem-HOWTO hasn't been
>   installed?  Is it active (and generates an error if it's clicked), or
>   is it inactive, or what? ]
> 
> > nor do they help a user find a document they've never seen before.  
> 
> That's what the index, table of contents, and free text search are for.


I agree - it will be a pain to support cross references properly (I wrote 
this in a previous mail but its stuck in the mail spool at home! DOH)

I think if people want "cross-links" they should use a 'See Also' at the 
end of the document

Regards,
Ali

Re: [Scrollkeeper-devel] Handling relocatable packages

[Gregory L. <gle...@cu...>]

> On Mon, Nov 27, 2000 at 03:42:38AM -0200, Ali Abdin wrote:
> > * Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
> 
> I haven't seen this sort of design document yet, and the "DD:" points in
> my original message were supposed to seed that, so that we can agree on
> some common functionality, irrespective of the specific implementation
> details.


Didn't somebody have these?  I could have sworn I'd seen them, but I
can't seem to figure out where.


> > > sk-generate-omf is used to generate OMF for installation into sk.  It can be 
> > > told about an existing OMF file (for example, if the document has been 
> > > distributed with an OMF file).  However, it also has a series of --omf:<foo>
> > > options, which are used to override the OMF in the .omf file, and replace 
> > > it with something else (I haven't listed them all in the synopsis above).
> > 
> > Umm - okay, lets say I have a well-formed OMF file and then I do
> > 'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?
> 
> Nothing.  But if you do
> 
>     sk-generate-omf --omf:creator 'Foo Bar <fo...@ex...>' file.omf
> 
> then sk-generate-omf has just rewritten the content of the CREATOR
> element in file.omf.
> 
>     sk-generate-omf < file.omf > file2.omf
> 
> is probably the more natural way of running it though.
> 
> > Hmm, you seem to be implying that people will not write OMF files and that
> > they should be "extracted" from the docs. From my understanding, they are a
> > part of the doc (there was discussion on wether to include necessary tags in
> > the docs themselves, or to have separate OMF files).
> 
> Not quite.  I think we need to cater for a few sets of eventualities.
> 
>   1.  No OMF provided, but document is in an OMF'able format, or where
>       some of the OMF can be extracted.
> 
>   2.  No OMF provided, and document is not in an OMF'able format (such
>       as plain text)
> 
>   3.  OMF provided, but it's incorrect in some way.
> 
>   4.  OMF provided, and it's correct.
> 
> In the case of (1) I think we need some tools that can extract OMF from
> common formats (DocBook, man, LinuxDoc, ...).  Whilst these tools are
> not strictly part of sk itself (anyone want to start an omf-tools
> project?) I think they'll be important to aid the take up of sk.


I've borrowed a perl hack from Greg Ferguson, and I've made (and had
made) a few changes to it.  However, there are a few things that don't
quite work just yet, I'll let people know when I've got tools working.


> In 1, we'll also need the sk-generate-omf command to update the OMF
> after it's been extracted from the source document.


Why does it need to be updated immediately after it's been
extracted?  Or did I miss something?


> In the case of (2) we'll need an sk-generate-omf command to make it easy
> for the end user to generate valid OMF files.
>
> [ To anticipate your question -- "Why not get the end user to create the 
>   OMF files using a text editor?", I think the reason is three-fold.
> 
>   First, the user doesn't need to learn a new text format.
> 
>   Second, this makes it impossible for the user to create an invalid OMF
>   file.
> 
>   Third, it insulates the end user from possible changes in the OMF file
>   format (i.e., the XML DTD). ]


I don't think this is something that should be part of sk, per say, but
some "extra" module.  


> In the case of (3), we'll need sk-generate-omf to fixup the incorrect
> OMF data.


That sounds like fun, I'm very familiar with "automagically" fixing XML
documents, based on the errors that a parser spits out.  I'd guess that
the w3c's Tidy application would contain some of the concepts needed for
this.


> > I think perhaps we should just allow the '--omf:' overrides in the
> > sk-install-omf binary
> 
> I disagree.  Don't have one tool that does multiple things, have
> multiple tools that do one thing each.  It's useful to be able to
> generate OMF without forcing the end user to install the documentation
> as part of the process.


Yes,  one tool that does one thing is "The Unix Way", and I think it's
probably a good way to go.  Would someone want to have this available at
the time that they install the package?  If so, it needs to be part of
sk.

    Greg

Re: [Scrollkeeper-devel] questions, questions

[Nik C. <ni...@no...>]

On Mon, Dec 04, 2000 at 01:48:45AM -0800, Bob Stayton wrote:
> I think the performance of XSL processing will improve
> within six to twelve months.  

Moore's Law says that we'll get double the speed in 12-18 months time
without doing anything.  Let's concentrate on doing things right first, and
then sort out hacks to speed up things if necessary later.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Mon, Nov 27, 2000 at 03:42:38AM -0200, Ali Abdin wrote:
> * Nik Clayton (ni...@no...) wrote at 03:22 on 26/11/00:
> > Just because the original implementors didn't provide a hook to make it
> > non-relocatable, don't assume that other's won't have done.
> 
> This is not scrollkeeper's problem though. We should support both instead of
> imposing relocatable packages on people.
> 
> Actually, I think if we supposed relocatable packages, we will (by definition)
> have supported non-relocatable packages, so this is a moot point.

Ah ha.  Someone gets it :-)  That's right exactly.

> > > Aside from rebuilding the database in cases where it becomes corrupt, I'd
> > > like this to be a zero maintenance system that works out of the box.  I
> > > don't think we should focus on unusual things a SA may want to
> > > do.  
> > 
> > "Tools, not policy" (see, I said I'd be coming back to it).  The things
> > that you are I think are unusual might be standard practice for someone
> > who wants to use sk.
> 
> I think the goal is to make scrollkeeper "automated" enough so that Desktop
> users can use it (remember - Help Documents are (usually) for new users).
> 
> Now if a sysadmin wanted to play around and mess with ScrollKeeper, I think
> this also should be possible. Now, what kind of things would a sysadmin want
> to do that should be supported?

It's not a "What would they want to do" thing.  It's a philosophy of not
building in restrictions (like "There will only ever be one contents
list") in to the system.  We might not be able to immediately see why
multiple contents lists would be useful.  But the extra effort in
supporting them is useful.

<snip>

> > but we should make it possible for the SA to change this behaviour if
> > necessary.
> 
> I really don't think this is a major issue (to specify which directory a file
> goes into). 

It's not a major issue, no.  What I'm trying to do is highlight where
people are (or could be) inadvertently hardcoding in design decisions
that haven't been thought through.

If we have a feature list that we agree with, and one of those features
is "sk will support multiple contents lists" then it's unlikely that
we'll forget that as we implement sk.

I haven't seen this sort of design document yet, and the "DD:" points in
my original message were supposed to seed that, so that we can agree on
some common functionality, irrespective of the specific implementation
details.

[ Quote kept for context ]

> > > > > Why do we specify the format on the command line?  The FORMAT is specified
> > > > > in the OMF already.  The packager should just install and register the
> > > > > metadata instead of putting it on the command line.
> > > > 
> > > > Because at this point, all we've done is install a document with no OMF.
> > > > SK has to synthesise some OMF, based on any additional information we 
> > > > give it.
> > > > 
> > > > This separation is necessary to support documents that have no separate
> > > > OMF.  Adding "-format" to the command line is probably going to be
> > > > easier for an end user than having to create a separate file to return
> > > > to.
> > > 
> > > Perhaps we agree on this after all and are just talking past each
> > > other.  So for docs with OMF we have two steps:
> > > 1) update the IDENTIFIER to point to the right URL
> > > 2) register the OMF file with sk
> > > 
> > > Really, #1 is only necessary for local docs (as opposed to remote
> > > ones) and for relocatable packages.
> > 
> > I think that's right.  In my example, "sk-install-doc" is probably the
> > wrong name.
> > 
> > Ah ha, I think I've got a work around.
> > 
> > We've been going back and forth about what to do if a document doesn't
> > have OMF, or if the doc does have OMF, but the URL in the INDENTIFIER is
> > wrong, or whatever.
> > 
> > So, what if we have two command, sk-generate-omf and sk-install-omf?  The 
> > usage for this would look something like;
> > 
> >   sk-generate-omf [--omf:creator ...]
> > 	  	  [--omf:maintainer ...]
> > 		  [--omf:contributor ...]
> > 		  [--omf:title ...]
> > 		  [--omf:date ...]
> >                   [<other --omf:... options]
> > 		  file.omf
> > 
> >   sk-install-omf [--omf-dir ...] /path/to/omf/file
> > 
> > sk-generate-omf is used to generate OMF for installation into sk.  It can be 
> > told about an existing OMF file (for example, if the document has been 
> > distributed with an OMF file).  However, it also has a series of --omf:<foo>
> > options, which are used to override the OMF in the .omf file, and replace 
> > it with something else (I haven't listed them all in the synopsis above).
> 
> Umm - okay, lets say I have a well-formed OMF file and then I do
> 'sk-generate-omf file.omf' - umm, what exactly has been "done" to this file?

Nothing.  But if you do

    sk-generate-omf --omf:creator 'Foo Bar <fo...@ex...>' file.omf

then sk-generate-omf has just rewritten the content of the CREATOR
element in file.omf.

    sk-generate-omf < file.omf > file2.omf

is probably the more natural way of running it though.

> Hmm, you seem to be implying that people will not write OMF files and that
> they should be "extracted" from the docs. From my understanding, they are a
> part of the doc (there was discussion on wether to include necessary tags in
> the docs themselves, or to have separate OMF files).

Not quite.  I think we need to cater for a few sets of eventualities.

  1.  No OMF provided, but document is in an OMF'able format, or where
      some of the OMF can be extracted.

  2.  No OMF provided, and document is not in an OMF'able format (such
      as plain text)

  3.  OMF provided, but it's incorrect in some way.

  4.  OMF provided, and it's correct.

In the case of (1) I think we need some tools that can extract OMF from
common formats (DocBook, man, LinuxDoc, ...).  Whilst these tools are
not strictly part of sk itself (anyone want to start an omf-tools
project?) I think they'll be important to aid the take up of sk.

In 1, we'll also need the sk-generate-omf command to update the OMF
after it's been extracted from the source document.

In the case of (2) we'll need an sk-generate-omf command to make it easy
for the end user to generate valid OMF files.

[ To anticipate your question -- "Why not get the end user to create the 
  OMF files using a text editor?", I think the reason is three-fold.

  First, the user doesn't need to learn a new text format.

  Second, this makes it impossible for the user to create an invalid OMF
  file.

  Third, it insulates the end user from possible changes in the OMF file
  format (i.e., the XML DTD). ]

In the case of (3), we'll need sk-generate-omf to fixup the incorrect
OMF data.

In the case of (4), we don't need to do anything.

> I think perhaps we should just allow the '--omf:' overrides in the
> sk-install-omf binary

I disagree.  Don't have one tool that does multiple things, have
multiple tools that do one thing each.  It's useful to be able to
generate OMF without forcing the end user to install the documentation
as part of the process.

<snip>

> > For documents that have no OMF (i.e., the author didn't provide one), 
> > sk-generate-omf can be used to generate one that SK can use.
> 
> I think at first, ScrollKeeper will only support DocBook SGML/XML (which I
> think is a decent goal for '1.0' (then from the experience with that we can
> possible tackle other (more difficult) file formats)).
> 
> If a document does not have an OMF file, we should not attempt to "guess"
> anything in any way. 

Agreed.  But tools that make it easier for the end user to have a stab
at generating OMF would be useful.  For example, it would greatly ease
the end user's job if they can do something like

    cd /usr/share/man/man1
    foreach i (*.1)
      man2omf $i > /usr/local/share/omf/man/$i.omf
    end

This doesn't mean that the generated OMF files are 100% correct (note:
not 'valid', as man2omf should always generate valid XML) -- for
example, the author information might say "Automatically generated by
man2omf", or some such.

As I say earlier though, this is not strictly sk's responsibility.
However, I suggest that development of small helper apps like these will
increase the speed with which people can usefully use sk on their
system.

> > The Contents List is in XML, right.  XML already has a mechanism for
> > saying that a file includes other files.
> > 
> >   <?xml version="1.0">
> >   <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
> > 
> >   <!ENTITY contents.bio  SYSTEM "/path/to/biology/contents/list">
> >   <!ENTITY contents.chem SYSTEM "/path/to/chemistry/contents/list">
> > 
> >   ]>
> > 
> >   <contents>
> >     &contents.bio;
> > 
> >     &contents.chem;
> >   </contents>
> > 
> > Why reinvent the wheel?  All we need to do is ensure (as reasonably as
> > possible) that the identifiers in the different contents lists are
> > unique.
> 
> Actually - this is a VERY good idea. 

Woo hoo

[ Sorry, too much Simpsons ]

> In fact, DV just suggested this idea on
> IRC which I didn't quite understand (then I read this example and I get it)...
> 
> Basically, you can have a /etc/scrollkeeper.conf file that lists all the
> directories which Scrollkeeper will scan and include as entities.

Not quite.  XML can do all this for you.

Suppose, for example, that the sysadmin wants to make it easy for end
users to create their own contents lists that include the system's
installed documentation.

They might create a default Contents List that looks like this:

   <?xml version="1.0">
   <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
 
   <!ENTITY contents.system  SYSTEM "/path/to/system/contents/list">
   <!ENTITY contents.user SYSTEM "&HOME;/.sk/contents/list">
 
   ]>
 
   <contents>
     &contents.system;
 
     &contents.user;
   </contents>

There's only one peice of 'magic' in here, "&HOME;".

Basically, what I'm trying to show here is that this master contents
list would pull in the contents of /path/to/system/contents/list, and
would then pull in /home/$USER/.sk/contents/list.

So if the user goes and creates their own contents list in
~/.sk/contents/list and runs sk then they're changes are automatically
picked up.

Unfortunately, XML (and SGML) has no concept of using environment
variables inside paths like this.

What you could do is hardcode sk to recognise $HOME inside a Contents
List and automatically expand it to the appropriate environment
variable.  However, that's a bit of a kludge, and also takes us outside
of XML functionality.

However, it's not too much of a stretch to say (and document) that when
SK parses a Contents List it pre-initialises a few entities for use in
the Contents List.  One of those might be &HOME;, which corresponds with
$HOME, another might be &LC_LANG;, which corresponds with $LC_LANG, and
so on.

Actually, that's a pretty cool idea.  You could imagine a master
Contents List that looks like

   <?xml version="1.0">
   <!DOCTYPE contents PUBLIC "-//Scrollkeeper Contents List//EN" [
 
   <!ENTITY contents  SYSTEM "/var/db/sk/&LC_LANG;/contents">
 
   ]>
 
   <contents>
     &contents;
   </contents>

which would pick up the contents list from /var/db/sk/$LC_LANG/contents
depending on the setting of the $LC_LANG environment variable of the
person running sk.

> I have a question though. How will this work for user's directories? Does
> Scrollkeeper need to support this "indepenendently (and track it with a
> ~/.scrollkeeperrc)

See above.  This approach actually lets the SA create a global config
file that automatically pulls in $HOME/.sk/, without the user needing to
do anything special.  The name of the file it pulls in is completely at
the discretion of the SA.

Of course, sk would ship with defaults that show how all this works.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Handling relocatable packages

[Nik C. <ni...@no...>]

On Mon, Nov 27, 2000 at 03:57:15AM -0200, Ali Abdin wrote:
> * Nik Clayton (ni...@no...) wrote at 03:49 on 26/11/00:
> > > > How?  libsk.so?  By having a program that accepts search parameters and
> > > > returns XML with the results?  Some other mechanism?
> > > 
> > > I prefer a library that returns the xmlDocPtr of the file...
> > 
> > xmlDocPtr is specifc to libxml, isn't it?  What if I (as an application
> > programmer) want to use another XML library?  Since (part of) the point
> > behind XML is to reduce reliance on one specific parsing library, it
> > would be more open-ended if libsk could be used to find a document,
> > which can then be referred to by URL.  It's then up to the calling
> > application to open that URL and process it.
> 
> ScrollKeeper /has/ to pick an XML library to rely on for parsing stuff. 

That's fine.  It just shouldn't force any other application that wants
to use the results from SK to use the same library.  The library might
not even be available on some platforms, or languages.  Suppose I'm
writing a wrapper around SK in Perl.  What do I do with a C pointer in
Perl?

SK is going to be used to identify documents in the catalogue, so it seems
natural to me that the result of any queries should be a URL (or list of
URLs) that point to the matching document(s).  Let the calling
application decide what it wants to do with this URL.

> I was
> just suggesting "publicizing" this in the API...Anyway, you /could/ have
> libxml-specific stuff trapped in #ifdef's and just provide "other" people with
> the FILE descriptor to do their own parsing.

Still too C specific. 

> Note: to compile (and run) scrollkeeper you will need to have libxml
> installed.

Maybe to compile it.  But you can always compile it static, and ship the
binary to a system that doesn't have libxml installed.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Simple URI's

[Nik C. <ni...@no...>]

On Mon, Dec 04, 2000 at 10:10:25AM -0600, Dan Mueth wrote:
> On Mon, 4 Dec 2000, Laszlo Kovacs wrote:
> > > Let the users use bookmarks.  On a network with multiple architectures,
> > > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> 
> Bookmarks won't help here.  Bookmarks do not provide a mechanism for one
> document to cross-reference another, 

Nor should they.  Document cross references are going to be a pain in
the backside, which ever way you cut it.  I don't think is really within
sk's purview.

[ For example, suppose I have a document that references modem-HOWTO in
  a link.  What happens to the link if modem-HOWTO hasn't been
  installed?  Is it active (and generates an error if it's clicked), or
  is it inactive, or what? ]

> nor do they help a user find a document they've never seen before.  

That's what the index, table of contents, and free text search are for.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] questions, questions

[Gregory L. <gle...@cu...>]

> * Bob Stayton (bo...@sc...) wrote at 16:38 on 04/12/00:
> >
> > I think the performance of XSL processing will improve
> > within six to twelve months.  If I were setting up a server
> > today, though, I would preprocess all my XML docs into HTML
> > files to be served.  Cron jobs or time stamps would have to
> > be used to keep the HTML current. This would be similar to
> > man pages being nroffed into cat files to improve
> > performance when Unix machines used to be slower.
> 
> Yeah, I asked Daniel Veillard about this. Note: Cocoon is in Java as far as I
> know and we all know about Java performance on Linux (I'm hoping a C
> implementation would show up so we can compare/benchmark it).
> 
> Daniel recommended for me to parse XML using DOM (until XSLT performance
> improves).


How does this get around the issues of XSLT being slow?  I'm probably
being obtuse, but I just don't see the connection.


> I wonder how the performance would be if soembody used a "stripped down"
> version of Norm Walsh's XSL stylesheet (I assume he supports a huge subset of
> tags (all of DocBook?) while the GNOME project only uses ~120 tags).


I'd much rather have the full set of DocBook tags available, as the
overhead of changing the stylesheets when Joe Author decides to use a
different tag is pretty high.  Does the manner in which XSL stylesheets
are written affect the speed of processing as much as it does for
DSSSL?  (One of the GNOME hackers had a REALLY awful DSSSL sheet the
other day, it took many minutes to run on an 18K file).

    Greg


P.S.  Hmm, we're probably drifting from the topics that should be on
sk-devel now...

Re: [Scrollkeeper-devel] questions, questions

[Ali A. <ali...@au...>]

* Bob Stayton (bo...@sc...) wrote at 16:38 on 04/12/00:
> > From: Gregory Leblanc <GLe...@cu...>
> > 
> > > From: Ali Abdin [mailto:ali...@au...]
> > > 
> > > Cocoon looks like an amazing project actually ;) But are 
> > > there some DocBook
> > > XSL's out there? (for DocBook -> HTML conversion) ?
> > 
> > Yep, they're available from Norm Walsh, at the same place as the DSSSL
> > stylesheets.  Hmm, URL...  Try http://www.nwalsh.com/docbook/xsl/index.html
> 
> I've used Norm's XSL stylesheets in Cocoon, and
> it is currently too slow to be acceptable for a
> web server.  His stylesheets are very big and take
> a considerable time to load.  In general, processing
> XML with XSL on the fly is not quick.  Cocoon relies
> on caching to give decent performance, but a document has
> to have been cached.  Browsers processing XML with XSL will
> have to work hard to not appear sluggish.
> 
> I think the performance of XSL processing will improve
> within six to twelve months.  If I were setting up a server
> today, though, I would preprocess all my XML docs into HTML
> files to be served.  Cron jobs or time stamps would have to
> be used to keep the HTML current. This would be similar to
> man pages being nroffed into cat files to improve
> performance when Unix machines used to be slower.

Yeah, I asked Daniel Veillard about this. Note: Cocoon is in Java as far as I
know and we all know about Java performance on Linux (I'm hoping a C
implementation would show up so we can compare/benchmark it).

Daniel recommended for me to parse XML using DOM (until XSLT performance
improves).

I wonder how the performance would be if soembody used a "stripped down"
version of Norm Walsh's XSL stylesheet (I assume he supports a huge subset of
tags (all of DocBook?) while the GNOME project only uses ~120 tags).

Regards,
Ali

Re: [Scrollkeeper-devel] Simple URI's

[Ali A. <ali...@au...>]

* Laszlo Kovacs (las...@su...) wrote at 16:36 on 04/12/00:
> > Let the users use bookmarks.  On a network with multiple architectures,
> > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> > 
> > If an individual user wants to create their own aliases for
> > documentation, they should probably be able to do so.  The user can then
> > have the burden of deciding which short name maps to which document, and
> > is in a much better position to know that they want "modem-HOWTO" to
> > point to the Solaris document.
> > 
> 
> Or the sysadmin decides where these URI's point to depending on machine
> type.

The "short name" should be settable by the sysadmin (for Desktop users, they
are the sysadmins). Each doc should have by default its own shortname.

But what should scrollkeeper do if it finds a conflict?

Regards,
Ali

Re: [Scrollkeeper-devel] Simple URI's

[Dan M. <d-...@uc...>]

On Mon, 4 Dec 2000, Laszlo Kovacs wrote:

> > Let the users use bookmarks.  On a network with multiple architectures,
> > does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?

Bookmarks won't help here.  Bookmarks do not provide a mechanism for one
document to cross-reference another, nor do they help a user find a
document they've never seen before.  Bookmarks are great, but they serve a
different purpose.

> > If an individual user wants to create their own aliases for
> > documentation, they should probably be able to do so.  The user can then
> > have the burden of deciding which short name maps to which document, and
> > is in a much better position to know that they want "modem-HOWTO" to
> > point to the Solaris document.
> > 
> 
> Or the sysadmin decides where these URI's point to depending on machine
> type.

We need to have a reasonable "default" document chosen.  For any given
document (ie a set of documents with different versions, languages, and
formats but usually the same author and title) we will need to default to
a reasonable choice.  (We probably also want a way to return them all to
choose from.)  This was in the original post.

The sysadmin will indirectly be determining where these URI's point to by
specifying the default COVERAGE.OS in /etc/sk.conf.  So sk will know to
return documents that apply to that OS (and in the right language, and for
the right distribution, ...).  That is why we have all this great metadata
- so the sys admin and user don't have the burden of doing everything by
hand.

Dan

Re: [Scrollkeeper-devel] Simple URI's

[Laszlo K. <las...@su...>]

> Let the users use bookmarks.  On a network with multiple architectures,
> does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?
> 
> If an individual user wants to create their own aliases for
> documentation, they should probably be able to do so.  The user can then
> have the burden of deciding which short name maps to which document, and
> is in a much better position to know that they want "modem-HOWTO" to
> point to the Solaris document.
> 

Or the sysadmin decides where these URI's point to depending on machine
type.

Laszlo

Re: [Scrollkeeper-devel] Simple URI's

[Nik C. <ni...@no...>]

[ Sorry for the delay in some of these, free time tends to come in
  discrete bursts for me ]

On Tue, Nov 28, 2000 at 01:47:29AM -0600, Dan Mueth wrote:
> GNOME uses a "gnome-help" URI scheme.  So if you type
> "gnome-help:gnumeric", it looks in
> $prefix/share/gnome/help/gnumeric/<locale>/ and looks for the doc under
> the name gnumeric.sgml or else index.html.  It is a very convenient way to
> find a manual.  It also allows for cross-referencing of documents very
> simply since any document you write knows that "gnome-help:gnumeric" will
> point to the Gnumeric manual regardless of where things are installed (eg.
> under /usr or /opt or ...).
> 
> The problem with this is:
> 1) other help browsers have to understand how to handle the
> "gnome-help" URI scheme

Big problem.

> 2) presumably other desktops will introduce more URI schemes

Wouldn't suprise me.  Probably without thinking about what others have
been doing in this space as well.

> 3) many docs won't be addressable at all this way unless scrollkeeper
> provides a mechanism.

True.

You missed

 4) The inevitable clash between two documents that want to have the
    same simple name.

> So probably what we want is to introduce a scheme which passes the
> identifier (what is the right term here?) to scrollkeeper which can return
> the path.  So for example, one would use: "sk-help:modem-HOWTO".  It
> would have to find the doc (and select from any available versions,
> languages, formats) and then display it.  This would be *very* useful IMO,
> both for users as well as for cross-referencing between documents.

Let the users use bookmarks.  On a network with multiple architectures,
does "modem-HOWTO" refer to the Linux, FreeBSD, or Solaris document?

If an individual user wants to create their own aliases for
documentation, they should probably be able to do so.  The user can then
have the burden of deciding which short name maps to which document, and
is in a much better position to know that they want "modem-HOWTO" to
point to the Solaris document.

N
-- 
Internet connection, $19.95 a month.  Computer, $799.95.  Modem, $149.95.
Telephone line, $24.95 a month.  Software, free.  USENET transmission,
hundreds if not thousands of dollars.  Thinking before posting, priceless.
Somethings in life you can't buy.  For everything else, there's MasterCard.
  -- Graham Reed, in the Scary Devil Monastery

Re: [Scrollkeeper-devel] Scrollkeeper backend

[Laszlo K. <las...@su...>]

> If we do decide that a db backend makes more sense, I'm not sure we would
> want to move over too soon.  It may be a good idea to work entirely in XML
> to keep things simple for now, until we have a solid version or two
> released and a development roadmap complete.
It would be very good if the chosen db backend would be easily available
on non-Linux platforms (like Solaris and other commercial Unix
platforms).

Laszlo

RE: [Scrollkeeper-devel] questions, questions

[Bob S. <bo...@sc...>]

> From: Gregory Leblanc <GLe...@cu...>
> 
> > From: Ali Abdin [mailto:ali...@au...]
> > 
> > Cocoon looks like an amazing project actually ;) But are 
> > there some DocBook
> > XSL's out there? (for DocBook -> HTML conversion) ?
> 
> Yep, they're available from Norm Walsh, at the same place as the DSSSL
> stylesheets.  Hmm, URL...  Try http://www.nwalsh.com/docbook/xsl/index.html

I've used Norm's XSL stylesheets in Cocoon, and
it is currently too slow to be acceptable for a
web server.  His stylesheets are very big and take
a considerable time to load.  In general, processing
XML with XSL on the fly is not quick.  Cocoon relies
on caching to give decent performance, but a document has
to have been cached.  Browsers processing XML with XSL will
have to work hard to not appear sluggish.

I think the performance of XSL processing will improve
within six to twelve months.  If I were setting up a server
today, though, I would preprocess all my XML docs into HTML
files to be served.  Cron jobs or time stamps would have to
be used to keep the HTML current. This would be similar to
man pages being nroffed into cat files to improve
performance when Unix machines used to be slower.

bobs
Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...

[Scrollkeeper-devel] Serving docs with SK

[Bob S. <bo...@sc...>]

> From: Dan Mueth <d-...@uc...>
> > 
> > Thanks for clarifying.  The application for ScrollKeeper I
> > have in mind is not using a browser directly, but serving
> > doc content with an HTTP server like Apache.  There are
> > several ways to use XML on a server (AxKit, Cocoon), so
> > using the XML Content List will be fine in that context.
> > The server can also convert content to HTML if it needs
> > to.  That approach can make ScrollKeeper-listed docs
> > available to any browser.  
> 
> This is a great idea. I've been thinking that the servers on the Net would
> be primarily be talking to help browser clients on the desktop and that
> the servers would just serve as metadata databases.  But coupling this
> with some conversion filters (SGML->HTML, SGML->PDF, SGML->PS, man->HTML,
> SGML->audio?, etc.) and giving it a very slick interface allows you to
> create a very nice help browser web application. For people with good
> internet connections, this would be a very useful tool.  It also gives you
> the benefit that you don't have to download an entire document since the
> server is converting the document to HTML - you just download the pages
> you read.  Also, any project, organization, company, etc. could use it
> with their own particular set of documents.  This would be a very good
> thing to pursue IMO.

Glad you think it is a good idea. I plan to pursue it.
That's why I want to make sure there is some flexibility
in accessing the ScrollKeeper data.

bobs

Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...

RE: [Scrollkeeper-devel] questions, questions

[Gregory L. <GLe...@cu...>]

> -----Original Message-----
> From: Ali Abdin [mailto:ali...@au...]
> 
> Cocoon looks like an amazing project actually ;) But are 
> there some DocBook
> XSL's out there? (for DocBook -> HTML conversion) ?

Yep, they're available from Norm Walsh, at the same place as the DSSSL
stylesheets.  Hmm, URL...  Try http://www.nwalsh.com/docbook/xsl/index.html
	Greg

Re: [Scrollkeeper-devel] questions, questions

[Ali A. <ali...@au...>]

* Bob Stayton (bo...@sc...) wrote at 02:07 on 02/12/00:
> > From: Dan Mueth <d-...@uc...>
> >
> > [stuff deleted]
> > 
> > As for just supporting GNOME and KDE, this is definitely *not* the
> > intention.  There are many people who do not use GNOME or KDE, and that
> > will always be the case.  I would be happy to see another help browser
> > which is independent of these two desktops and which can utilize
> > scrollkeeper.  If there was a compelling reason to use some format other
> > than XML, then scrollkeeper's API could be extended.
> 
> Thanks for clarifying.  The application for ScrollKeeper I
> have in mind is not using a browser directly, but serving
> doc content with an HTTP server like Apache.  There are
> several ways to use XML on a server (AxKit, Cocoon), so
> using the XML Content List will be fine in that context.
> The server can also convert content to HTML if it needs
> to.  That approach can make ScrollKeeper-listed docs
> available to any browser.  

Cocoon looks like an amazing project actually ;) But are there some DocBook
XSL's out there? (for DocBook -> HTML conversion) ?

Regards,
Ali

Re: [Scrollkeeper-devel] questions, questions

[Dan M. <d-...@uc...>]

On Fri, 1 Dec 2000, Bob Stayton wrote:

> > From: Dan Mueth <d-...@uc...>
> >
> > [stuff deleted]
> > 
> > As for just supporting GNOME and KDE, this is definitely *not* the
> > intention.  There are many people who do not use GNOME or KDE, and that
> > will always be the case.  I would be happy to see another help browser
> > which is independent of these two desktops and which can utilize
> > scrollkeeper.  If there was a compelling reason to use some format other
> > than XML, then scrollkeeper's API could be extended.
> 
> Thanks for clarifying.  The application for ScrollKeeper I
> have in mind is not using a browser directly, but serving
> doc content with an HTTP server like Apache.  There are
> several ways to use XML on a server (AxKit, Cocoon), so
> using the XML Content List will be fine in that context.
> The server can also convert content to HTML if it needs
> to.  That approach can make ScrollKeeper-listed docs
> available to any browser.  

This is a great idea. I've been thinking that the servers on the Net would
be primarily be talking to help browser clients on the desktop and that
the servers would just serve as metadata databases.  But coupling this
with some conversion filters (SGML->HTML, SGML->PDF, SGML->PS, man->HTML,
SGML->audio?, etc.) and giving it a very slick interface allows you to
create a very nice help browser web application. For people with good
internet connections, this would be a very useful tool.  It also gives you
the benefit that you don't have to download an entire document since the
server is converting the document to HTML - you just download the pages
you read.  Also, any project, organization, company, etc. could use it
with their own particular set of documents.  This would be a very good
thing to pursue IMO.

This may in fact have been the part of Metalab's original plan.

Dan

Re: [Scrollkeeper-devel] Scrollkeeper backend

[Bob S. <bo...@sc...>]

> From: Dan Mueth <d-...@uc...>
> 
> There is definitely room in the design for this.  I do not have a good
> idea how well the current implementation will scale, and it is entirely
> possible that we will reach a point where performance becomes an issue
> which drives us to another backend.  Aside from performance, the querying
> features of a database may be useful as well.  What else would we gain?

Those are the two main advantages.
It might also make building that sk API easier, depending
on what direction that design goes.
 
> I would be interested to hear more about the options.  I think we
> definitely would need it to be Free, lightweight, and in common use.

Understood.
 
> If we do decide that a db backend makes more sense, I'm not sure we would
> want to move over too soon.  It may be a good idea to work entirely in XML
> to keep things simple for now, until we have a solid version or two
> released and a development roadmap complete.

Understood, completely. 8^)
 
bobs
Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...

Re: [Scrollkeeper-devel] questions, questions

[Bob S. <bo...@sc...>]

> From: Dan Mueth <d-...@uc...>
>
> [stuff deleted]
> 
> As for just supporting GNOME and KDE, this is definitely *not* the
> intention.  There are many people who do not use GNOME or KDE, and that
> will always be the case.  I would be happy to see another help browser
> which is independent of these two desktops and which can utilize
> scrollkeeper.  If there was a compelling reason to use some format other
> than XML, then scrollkeeper's API could be extended.

Thanks for clarifying.  The application for ScrollKeeper I
have in mind is not using a browser directly, but serving
doc content with an HTTP server like Apache.  There are
several ways to use XML on a server (AxKit, Cocoon), so
using the XML Content List will be fine in that context.
The server can also convert content to HTML if it needs
to.  That approach can make ScrollKeeper-listed docs
available to any browser.  

bobs
Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...

Re: [Scrollkeeper-devel] Scrollkeeper backend

[Dan M. <d-...@uc...>]

On Thu, 30 Nov 2000, Bob Stayton wrote:

> > From: Ali Abdin <ali...@au...>
> 
> > > You just contradicted yourself and confused me. If the XML is a purely
> > > internal data structure, why is Nautilus reading it?
> > 
> > Yes, I did contradict myself :) 
> > 
> > Its not really an internal data structure or file format.
> > 
> > It is an external one, and instead of devising our own, we used an easily
> > parsable existing one...XML. I mean if you really want we could use a fast
> > binary database, but I doubt people would appreciate that ;)
> 
> I completely agree that XML is a good standard format to export
> data to applications like a help browser.  And of course
> the incoming data is XML (the OMF files).  
> 
> But I think there is room in the current design for a
> backend using a fast binary database to house the metadata.
> Whether it will be appreciated depends entirely on the
> applications that might use it.  Scrollkeeper could be used
> to keep track of metadata for thousands of documents.
> Those documents might be served upon request by an HTTP
> server, after a query engine processes a metadata request.
> You won't want to scan a big XML metadata document for each
> such request.

There is definitely room in the design for this.  I do not have a good
idea how well the current implementation will scale, and it is entirely
possible that we will reach a point where performance becomes an issue
which drives us to another backend.  Aside from performance, the querying
features of a database may be useful as well.  What else would we gain?

> I'm thinking that Scrollkeeper could maintain an internal
> lightweight database of the OMF data.  A database is easy
> to update when one OMF file added, changed, or deleted. And you
> can extract an XML Content List in various forms as
> queries and reports from the database.  The source of the
> data will always be the original OMF files, so you can
> still rebuild the database from scratch.
> But a database would give sk greater flexibility and
> performance in making the metadata available to different
> applications.
> 
> I'm not volunteering anyone else's copious free time
> to do this.  8^)
> If the group is interested, I'd be happy to 
> pursue this area a little further.  There are several
> available technologies that might fit the bill.

I would be interested to hear more about the options.  I think we
definitely would need it to be Free, lightweight, and in common use.

If we do decide that a db backend makes more sense, I'm not sure we would
want to move over too soon.  It may be a good idea to work entirely in XML
to keep things simple for now, until we have a solid version or two
released and a development roadmap complete.

Thanks for your offer to help :)

Dan