Re: [Scrollkeeper-devel] ScrollServer

[Dan M. <d-...@uc...>]

On Thu, 6 Sep 2001, David Merrill wrote:

> Greetings ScrollKeeper Hackers,
>
> I'm putting together a Python based front end to the ScrollKeeper
> database which I'm calling ScrollServer. It is in a preliminary stage
> but anyone who wants to look at it is welcome to.
>
> ScrollServer loads the extended contents list into an object
> hierarchy, and then provides an HTML interface to the documents using
> the SimpleHTTPServer class which comes with Python.
>
> The idea is that rather than needing the Gnome Help Browser or
> Konqueror to get to the database, you can use any browser, including
> Lynx at the command line.
>
> Today, it only supports a single view (a hierarchical list of the
> documents by their section) and local html files.

This sounds like a really great project.   I'm happy to see somebody
working on this already :)

> Questions:
>
> Do you plan to prepend file:// to the source file location?

Yes.  I think we left this out in early releases but I believe it was
added at some point - perhaps the development version in CVS.

> When do you plan to support http:// (remote files)
>
> Do you plan to support ftp:// at any point?

This is not really high on my personal list of priorities.  However, I
would expect that anybody who wants to add this could do so at any time.
It certainly makes things more interesting and powerful.  It also makes
things a bit more complicated because you have to consider the behavior
when docs dissappear or one's entire internet connectivity goes down.

> When will you add man pages?
>
> When will you add info pages?
>
> Do you have, or wish to have, a script that searches for man/info
> pages and adds them to the database? If so, how should it work?

I'm still unsure of how we want to deal with man and info pages.
ScrollKeeper doesn't have a whole lot to add here, since I doubt anybody
will want to write up OMF files for every man and info page out there.
I am guessing we will want to use some sort of a script which looks at
certain paths and extracts the info naturally embedded in these formats.
While this will be much more limitted than what we can do with
DocBook/OMF, I think it will still be a nice feature.  This is another
really good project for any hacker who is interested.

> How would you feel about a project whose package did nothing except
> install meta-data into the SK database? I'm thinking I could provide
> links to all the LDP docs in the local database just by having a
> `package' (.deb on debian) which adds the meta=data.

This is an interesting interim idea.  Eventually we would want to have the
metadata ship with the docs.  In the mean time, this sounds like a fine
idea.  I'm not sure if it is much less work than just making new packages
of the docs.

As soon as we support remote documents (http://, ftp://), it may make a
sense to have special packages of metadata for remote documents.  For
example, we could have ldp-remote-docs-1.noarch.rpm which just has a bunch
of OMF files which point to documents on the LDP web server.  I'm not sure
how well this works though since SK would have to download them all in
order to create the index files and other generated metadata files.  It
might work better to have remote metadata servers which your local machine
communicates with, just passing the metadata back and forth until the user
identifies which parts of which documents they want to read.

> Would you like to include the LDP meta-data in SK as it is shipped?

I don't see how this is possible, since the location(file://???), version,
existance, etc. of the file will vary with distribution.  I guess the FHS
helps us in terms of its location, but version and existance may still be
a problem.  I think shipping the metadata with the documents is the best
approach.

> Do you think any of this functionality, specifically the Python based
> object hierarchy, belongs as part of ScrollKeeper itself?

I'll assume "no" until you illustrate the contrary ;)

> Do you plan to implement search functionality, or is that the
> responsibility of each browser?

I think we would eventually like to support searching in ScrollKeeper,
although it will probably mean we use some existing text indexing program
like medusa to do all the work.  I think the reason why it belongs in (or
under) ScrollKeeper is that when we eventually have the ability for two
ScrollKeeper systems to communicate and share metadata, we will want one
server to ask the other to do a search and not to pass all the documents
across the net to do the search.

I'm not sure how well-suited medusa is for this task... I've been told it
would not need too much modification to accept a list of files to index.
I'm also not sure what other indexing apps are out there.  Hmmm...
Finding the right text indexing tool and hooking it into ScrollKeeper
seems like yet another project for a new hacker who wants to help out :)

Of course I'd *really* like to see all the functionality for ScrollKeeper
0.3.x get worked out and tested, and a stable 0.4.0 come out.  The new
features are really a huge improvement over 0.2 but need a lot of testing
and possibly a fair amount of work yet.  (Care to comment on the current
status Laszlo and Mary?)

Dan