[Scrollkeeper-devel] system similar to ScrollKeeper

[Bob S. <bo...@sc...>]

This mail describes the ODSSI (Online Documentation System
Services Interface) system that I mentioned in an earlier
mail. I'm sending this because ODSSI has goals similar to
ScrollKeeper.  Let me be clear that this is *not* a
proposal for ScrollKeeper, nor I'm not expecting any
reviews of our system (believe me, with three years
of experience using it, I'm well aware of its
weaknesses 8^). This mail is simply a description of one
actual implementation of a similar system.  I'll present
what we did, and then describe how I think it relates to
ScrollKeeper.  You can skip down to "How this relates
to ScrollKeeper" if you don't want all the details.

The SCO Help System
-----------------------

The SCO help system consists of an HTTP server delivering
HTML and man page content.  Content can be browsed locally
or remotely using any modern browser (see
<http://uw7doc.sco.com> for an example).  Local browsing is
also done through the HTTP server, using http://localhost.
Our first release used a customized browser, but we
abandoned that approach because once the system is on a
network, people wanted to use other browsers.  So we moved
all special processing to the server side.

Where's the doc?
-----------------------

The help server runs on a dedicated port and all doc is
installed below its DocumentRoot into language subtrees.
Under each language is a flat directory structure, one
"book" per directory.

DocumentRoot
  |
  |- en
  |   |- DesktopGuide
  |   |- ModemSetup
  |   |- NetworkConfig
  |   |- PortingGuide
  |   |...
  |- fr
  |   |- DesktopGuide
  |   |- NetworkConfig
  |   |...               [not all content is translated]

Here "book" is taken to mean a collection of html files
under one directory. The directory name also serves as a
unique book ID.  A book is a coherent unit of documentation
that should stand on its own, although it may reference
other books. It can be a real multi-chapter book, or a
short topic like a HOWTO.  Most of our content shifted from
big books to small single subject booklets, more like
HOWTOs.  Most of our content is also "chunked" into many small
HTML files for quick response.

If a book has more than one chapter, the book TOC lists the
chapter TOCs, each of which in turn lists its individual html
files.  For updates, we ship a new chapter TOC if a chapter
has changed, and the installation tools can rebuild the
main TOC if we add or remove a chapter.  Or a book can
simply supply a single fixed TOC and replace that in an
update.

Sometimes we use symlinks to link in content from other
parts of the filesystem so it becomes "visible" under
the DocumentRoot of the HTTP server.   Another exception is
man pages, which are installed in their usual location
/usr/man.  Man pages are accessed through CGI to get at
them. The CGI script also lets the system match the
functionality of command line man to find a man page
(section selection, apropos, MANPATH, etc.).


Content management goals 
---------------------------

These are the goals of the help system and ODSSI:

- Serve books, smaller topics, and man pages.
- Support multiple languages, with fallback using content negotiation.
- Support additional new content, from SCO and third parties.
- Support incremental page additions, replacements, deletions.
- Uninstall content cleanly (leave no traces).
- Organize the content into subjects for presentation.
- Regenerate navigational list pages when content changes.
- Regenerate search indexes.
- Minimize the burden on content providers.
- Support content with incomplete metadata.
   
Doc installation
---------------------------

Packages are expected to install doc under the DocumentRoot
location. If that isn't feasible because of other
requirements, the package needs to symlink the content
to appear under DocumentRoot, otherwise the HTTP server
cannot reach it.  

Each package installing documentation is expected to
also install three metadata files along with its doc content.
Then the package postinstall script is expected to call three or
four help configuration scripts already resident on the system.
These four scripts comprise the Online Documentation 
System Service Interface (ODSSI):

config_book   -- configures a book
config_man    -- configures a man section
config_views  -- updates the navigational pages
config_search -- updates search indexes

Configuring a book
---------------------------

This is what config_book does for each book.
If it is called without a bookid argument, it
processes all books that need processing.

1.  Determine the list of languages to process
from the language option, or default to English.

2.  For each language, set the LANG 
environment variable (this is necessary to get
proper sorting of titles if that is called for).

3.  cd to the book directory, and read all the *.node metadata
files to assemble a list of active HTML filenames and their
Title fields.  If there are no *.node metadata files, the
fallback is to collect the list of filenames and <TITLE>
elements from within the HTML files in the directory.

4.  If the *.book metadata file permits it, rebuild the
book table of contents.

5.  Generate the file list used by the search indexer.

The idea of "active" files came from a packaging limitation
that made it impossible for an update package to delete
an obsoleted file owned by another package.  Instead, we ship
new metadata for a file with status field marked as "delete".
Then that file is excluded from further processing
and taken out of the search index.  It just hangs
around on the system until its original package
is removed.

To regenerate the book TOC, the script looks in the *.book
metadata file, which identifies the type of TOC:

alpha   - book TOC is an alphabetical list of titles.
chapter - book TOC is a list of chapter titles.
fixed   - book TOC is fixed and may not be regenerated.

An "alpha" TOC is regenerated from the sorted list of Title
fields for the active files.  This TOC style is used for
glossary and encyclopedia type material, and man sections.

A "chapter" style TOC is rebuilt using metadata from the
*.node file that identifies active chapter TOC files.  When
a chapter is updated, we ship a new chapter TOC file.  The
rebuild of the book TOC lets us add, delete, or rearrange
the sequence of chapters in the book.  The individual
chapter TOCs then handle the rest of the section
hierarchy.  Why not just ship a new book TOC?  Because a
book can be updated from more than one independent update
package, and we don't want to overwrite other 
updates.  A TOC type of "fixed" is not regenerated.

The search indexer filelist for each book is kept on the
system. When a book is reindexed, the indexer first removes
all index records for files in the old list, and then
indexes all the files in the new list.  This lets us
eliminate obsolete files from the search index without
having to delete the whole index and reindex the whole doc
set.  

Book metadata
---------------------------

There are two metadata files per book that support the
above actions:

- A *.node file provides per-file info for a collection of files.

- A *.book file provides book-level info (title, bookid, etc).

The node file has one record for each HTML file. Here are three
sample records from a node file:

MouseOverview.html:
        Title = Overview of the mouse software
        Version = 1.0
        PrintSequence = /HW_mice/CTOC-Mouse_Administration.html

AddingMouse.html:
        Title = Adding a mouse device
        Version = 1.0
        PrintSequence = /HW_mice/CTOC-Mouse_Administration.html

CTOC-Mouse_Administration.html:
        Title = Mouse administration 
        Version = 1.0
        ChapterTOCWeight = 3

The first two records are for ordinary HTML content.
They both have a "PrintSequence" field that points to
the third record, which is a chapter table of contents
file (CTOC) of which they are a part.

A book can have several *.node files, typically an original
and one or more updates. If two node files contain metadata
for the same HTML filename, then the record with the higher
version number is used.  That permits the metadata to be
updated without replacing the original node file.  If that
higher version number includes a "Status = delete" field,
then that HTML file is dropped from the active list as
described above.

The "ChapterTOCWeight" field of the CTOC record identifies
that file as a chapter TOC, and suggests where its entry
should appear in the regenerated book TOC.  The weight
fields are sorted numerically for the book TOC.
The numbers can be decimals (e.g., 3.1) to permit
a new chapter to be inserted between two existing chapters.
An update package can drop a chapter without deleting it
by including a new record with "Status = delete" for
the CTOC file.

Each CTOC file is a typical nested list HTML file
presenting the chapter section titles. But it also contains
parsable metadata for that chapter hierarchy, identifying the
sequence and section level of all the HTML section files.
That metadata makes it easy for a CGI script to assemble
a complete chapter for printing, a feature users like.
It doesn't have to parse the HTML markup to figure it out.

We don't try to regenerate chapter TOCs after independent
updates because of the hierarchy.  Independent updates
of a simple list like the book TOC chapter list are easy,
but independent updates of a complex hierarchy can lead
to a scramble.

Man sections are handled by a separate config_man script.
That script generates an alpha TOC for each man section,
scanning over all the MANPATH directories. It also extracts
NAME lines to build the man apropos database.  It can
handle man pages in three formats: man, cat, and HTML.

Configuring the site
---------------------------

The above process manages just the content within a book
directory or man section. There is another script
"config_views" to manage how each book fits into the mix of
content on the server.

In our first design, we divided up the domain of Unix OS
information into what we thought was a complete subject
hierarchy.  We would be fitting each book into that fixed
hierarchy.  It didn't work.  New categories emerged as the
industry progressed (or we discovered them).  And some
content was hard to fit into one location in the
hierarchy.  I like to use the example of networked
printers.  Does that go under "Printers" or "Networking"?
Users are likely to look under either, so it needs to be
under both.  Which meant we either repeated information in
several books, or we devised a way to reuse information.

It became clear that there are several ways to "slice"
and present a large collection of information, depending
on your point of view (developer, administrator, end user).
Certain subjects could be handled by simply pointing at
a specific book.  But others needed to draw on parts of
other books, and even selections of man pages.  

So we developed the idea of "views" as a layer on top of the
content.  A view is a listing of content drawn from various
points in the doc collection.  Each view is a flat list of
view records.  An individual view record is an HREF that
can point to another view file, a book TOC, a chapter TOC,
a man section TOC, or an individual HTML file.  As you follow
a sequence of links, you might pass down through two views,
a book TOC, a chapter TOC, and then land on your desired file. 
Another sequence might be just two views and then a man page.

You might think this is a hopeless mess, mixing categories
and content willy nilly.  But the user sees it as the
shortest path to what they are looking for (one hopes).
The views are user oriented, while the content is 
package oriented.  You can draw an analogy with
documentation that sequentially covers all the menu
options in a program.  It is highly organized and complete.
But it doesn't much help a user who needs to put together
items from four different menus to accomplish a task. 

This design permits the creation of a view hierarchy that
supports multiple presentations of the same material,
organized in different ways.  For example, we present
a system Handbook that has no chapters of its own, but is a
collection of selected chapters from many books.
You can see this in action at http://uw7doc.sco.com,
scroll down and select "Handbook" from the left frame list.

Also, by using separate view
metadata, it lets you update or ship whole new view
branches after the doc itself ships.  Administrators
could even write their own views, similar to creating a 
bookmark hierarchy.

The other major feature of the design is that the view data
is modular, not monolithic.  The complete view hierarchy is
assembled from a lot of small contributions provided by
the packages.  Each package provides information about its own
content and "hints" for how it should be presented.

View metadata
---------------------------

Each book includes a *.view metadata file that contains one
or more view records that pertain to that book's content.
A single record of view metadata can have these fields:

viewID:
      Title = hot spot text
      URL = hot spot URL
      ParentView = viewID that contains this hot spot
      [Frame = framename]
      [Class = classname]
      [Weight = numerical sort of arbitrary numbers]
      [OverviewURL = URL for intro text]
      [Version = version number for this viewID]

View records are really of two kinds: those that point to
existing HTML files (or TOCs), and those that generate
new view pages.  

viewID
   A unique string that identifies a view record.
   The string can include "/"s, so they typically
   contain some path info to make them unique.

   A piece of doc can have more than one view record
   associated with it, with different viewIDs. That
   enables the doc to show up in more than one place
   in the view hierarchy.

Title and URL
   These are used to form the hyperlink:
      <A HREF="URL">Title</A>

ParentView
   The page that contains this hyperlink.
  
Frame
   Optional field to add TARGET="framename" to the
   hyperlink.
   
Class and Weight
   Are used to group and sort view records that appear
   on the ParentView.  All view records on a view page
   with the same Class name are grouped together between
   <HR> lines.  This partitions the list into subsections.
   The null Class name appears as the first group, with
   other group appearing in alpha order of Class name.
   Within a given Class name, view records are sorted by
   the numerical weight values.  "Weightless" records float
   to the top.  Finally, all records with
   the same Class and Weight are sorted alphabetically.
   Thus if no Class or Weight information is given for any
   records on a view page, you get a straight alphabetical
   list on that view page.

OverviewURL
   For a view record that defines a new view page, this URL
   points to content that explains the view page.
   The <H1> on the generated view page is made into a
   hyperlink that points to this URL.  In a frames-based
   system, this content is also displayed in the right
   frame while the view page is displayed in the left
   frame.

Version
   Version of the view record.  If more than one record for a
   given viewID exists in different *.view files, only the
   highest versioned record is used.  This lets you
   supercede a view record without deleting the original,
   which might be in a different package.

The help system ships with a set of top level view records.
The help home page has viewID "topview", and its ParentView
is "ROOT".  No other page can have parent ROOT.  A set
of basic categories are shipped with ParentView = topview
to populate the home page with the likely suspects of
the basic installed doc set.  The rest of the hierarchy
comes from the doc packages *.view files.

View processing
---------------------------

When a doc package is installed, the postinstall script
executes config_views.  In the current version, the entire view
hierarchy is rebuilt each time.  Here is what happens:

1.  Processes English first, and then other languages.  This
assumes the English doc set is complete, and is used to
fill in the gaps in the translated views with untranslated
content.  This ensures that all languages have access to the
same set of information, even if some is in English.

2.  Set LANG for the current language (to ensure proper
sorting of titles).

3.  Run a "find" command under DocumentRoot/<LANG> to locate
all *.view files for that language, and load the
data into associative arrays.

4.  For duplicate viewID records, keep only the record with
the highest Version number.

5.  Make sure a viewID exists for every ParentView trying to
reference it.

6.  Check the URLs to make sure they exist.  Any view record
whose URL does not exist is ignored.  This ensures that all
view links work.

7.  Sort the view records by:
     parent
     class
     weight
     title 

8.  For each new parent, generate an HTML view page using
its title and writing to its URL.  If it has an OverviewURL,
make its <H1> a hyperlink to the OverviewURL.

9.  For each view record for that page, generate a <A HREF=...>
hyperlink.  Draw <HR> when Class name changes.  

10.  Repeat for other languages.


Note that the view database is not kept, only the
generated HTML view files derived from it.  The processed
view records exist only in memory until config_views
exits.  This ensures that the view data cannot get out of
synch with a persistent database.  Also, this is the only
time the view data is used, and it only takes a few minutes
to regenerate all the views.  If this data had other
purposes it would be written out to disk.

How this relates to ScrollKeeper
---------------------------------------

Like I said, this system is not a proposal for ScrollKeeper
because of differences like these:

- All doc content resides under a DocumentRoot and is served with
  an HTTP server.

- Most of the content comes from one vendor (SCO), so there
  is a lot of coordination between packages.

- Big emphasis on handling incremental updates.

- Only serving HTML and man page document types.

- No persistent database of content.

But I think there were some lessons learned that do apply to
ScrollKeeper:

1.  A predetermined complete categorization of content is
not useful or even possible.  Rather, an open framework
of general categories can be established so providers
have a place to hang their stuff.  But providers need
the ability to add subcategories.

2.  There's more than one way to organize and
present the content.  Multiple different views can
be layered on top of the content collection.  This requires
looking at the content from the point of view of the multiple
different users, not just the content provider.

3.  Minimize the requirements on providers.  They should
just need to provide standard metadata and then call a single
post-install script already available on the system.
That script would take various actions based on the
metadata.  In a corollary, minimize the demands on
system administrators, since they generally have little
time to manage documentation.

4.  The system needs to handle content that has incomplete
metadata.  Its degree of participation and integration into
the system may be reduced, but it should not be excluded.
Our system went even further and processed content with
no metadata and no calls to the postinstall scripts, but
that's because we could locate all the content under
DocumentRoot.

5.  Metadata can be versioned, so new metadata can supercede
old without installing new versions of the doc.  You can
even mark some doc as obsolete without actually removing it.

6.  The system needs to take into account native language
translations, and provide a system of fallback when some
content is not available in a given language.  

7.  Clean removal of content is very important, and harder
than you think.  The removal scripts need to clean up
databases, views, and search indexes.  We didn't pay enough
attention to this item, and most of our bug reports came
after some content was incompletely removed.

I think many of these items have already been proposed
and discussed in the design so far.  

Conclusions
-----------------
Based on my experience to date, and at the risk of stepping
on someone's toes, I suggest that ScrollKeeper
divide its efforts into two areas:

1.  Inventorying doc content on a system.

2.  Presenting that content in various ways.

The inventorying effort includes defining the OMF metadata,
creating a database to house the data, and defining an API
for accessing the data.  This effort could also include
methods for finding non-registered doc (for example,
scanning RPMs for documentation sections), and fallback
processes for discovering and filling in missing metadata
for various file types.  The goal here is to define standard
methods for recording and reading the metadata.

The presentation effort can go in many different directions
to meet many different needs.  I don't think the
ScrollKeeper project can anticipate all the uses for a
documentation database like this, nor can it develop all
applications that use it.  One site will want to use a
Verity search engine to integrate doc with their other
searchable content, while another site will want to create
a website of documentation and index it with htdig, and a
third will load all the information into their existing
object-oriented database.  If there is a standard API for
extracting information from the ScrollKeeper database, then
those applications become much easier to do.

I do think the ScrollKeeper project should develop one
or more basic presentation applications, to the degree that it
can.  This will make it useful "out of the box" and
encourage content providers to add OMF data to their packages.
It will also test the data and the API, and lead to 
incremental refinement of both.

I think the ScrollKeeper project is already doing most
of this. I would just like to see a clear distinction
between forming the database and developing the applications
from it.  It would make it easier to divide up the work,
and clarify what needs to be delivered.

Of course, this is all just my opinion.  8^)

bobs
Bob Stayton                                 400 Encinal Street
Publications Architect                      Santa Cruz, CA  95060
Technical Publications                      voice: (831) 427-7796
The Santa Cruz Operation, Inc.              fax:   (831) 429-1887
                                            email: bo...@sc...