Thread: [Scrollkeeper-devel] ScrollKeeper-0.1 Proposal #3

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi everybody,

I've been sitting on this for a few days, hoping I will have time to
polish it up and fill in a few missing parts.  Things have gotten very
busy for me though and the mailing list has gotten more active lately, so
I think it is best I just send this so everybody can see where we are at.

The main difference between this proposal and proposal #2 is the
installation method.  We originally passed an argument to the installation
script which would announce where the new OMF files are at.  The benefit
was that the OMF file could be installed anywhere (potentially even on a
different computer) and that ScrollKeeper would know exactly what to
install/uninstall without wasting any time finding out by other means.  
The downside was that it made things somewhat complicated for developers
and packagers, since they had to prepare their metadata before
installation and pass more variables around.  After talking with several
people about this, we were "strongly encouraged" to abort our previous
plan and have all of the OMF files installed in a central location and
then have the install script detect changes in this directory.  The speed
penalty does not seem significant according to our tests, so I think the
benefit in terms of simplicity to developers and packagers as well as in
robustness against inevitable errors made by packagers outways the
slowdown.

Feedback is welcome :)

Dan

-----------

                --------------------------------------
                |   ScrollKeeper 0.1 - Proposal #3   |
                --------------------------------------

This document is arranged as follows:
  I.   Definitions and Overview
  II.  Building, Installing, and Uninstalling
  III. Other Tools for Admin and Packaging
  IV . ScrollKeeper Internals
  V.   Example File Snippits

I. Definitions and Overview

Contents List - This is a list of documents on the system (or more
generally, including docs which are not local to the single machine),
sorted into a tree.  Generally the tree will be sorted by subject, probably
using a controlled list of subjects.

TOC (Table of Contents) - This is a tree representing the sections and
subsections of a document.  The TOC can be extracted from an SGML/XML file.

Extended Contents List - This is the Contents List with the TOC for each
document merged into it.  Thus, one can not only navigate through the
the docs on the system, but also the sections within each doc, all in
one tree.

Index - This is a list of terms/concepts with links to parts of a document
discussing each of the terms.  This is often found at the back of printed
books.  In DocBook, indexing information is manually inserted into the
document by an indexer using indexing markup. Thus, an Index can be 
extracted from an SGML/XML file.

Generally a help browser will want to present all of the above information
to the user.  The Contents List is determined from OMF metadata, either
from a standalone file or from a DocBook doc.  The TOC and Index are generated
directly from the doc.  Since all of this is metadata and must be accessed
rather frequently, and to avoid potential performance problems, this 
metadata will be processed only at install and uninstall time.  The 
data will be stored in XML files for easy access by the help browser as
necessary.

II. Building, Installing, and Uninstalling

i. Building

The OMF documents are manipulated during the application
build process, generally as part of 'make all'.  The purpose of this
is to (1) substitute the correct URL of the document where it will
actually be installed, which is not generally known until build time,
and to (2) extract the OMF from DocBook documents into a seperate file.
(NOTE: #2 may not be supported in ScrollKeeper 0.1)

This processing is done by 'scrollkeeper-preinstall'.  This generally
takes three parameters: the URL for the installed document, the 
original OMF metadata file in XML, and the name that the generated 
OMF file should be written to. For example:

all:
	<stuff>
	scrollkeeper-preinstall file:$(DOCDIR)/foo-manual.sgml foo-manual-fr.omf omf-install/foo-manual-fr.omf 

If the path where the generated OMF file does not exist, it is created.  
So in this example, the IDENTIFIER in foo-manual-fr.omf is substituted with
a file URL which will point to the installed document and the 
subsequent OMF file is written in a new directory called "omf-install".

Note that the OMF file is named <application>-<document_title>-<locale>.omf.
This is important to avoid collisions between OMF files when they are
installed. 

ii. Installing

Installation is done by first copying the document and OMF file into
place and then having ScrollKeeper update its database by calling 
'scrollkeeper-update'.

install:
	install foo-manual.sgml $(DOCDIR)
	install omf-install/foo-manual-fr.omf $(OMFDIR)
	scrollkeeper-update -p $(SCROLLKEEPER_DB_DIR)

$(SCROLLKEEPER_DB_DIR) is the directory to use as the ScrollKeeper
database directory.  If $(localstatedir) is set during the configure 
stage, $(SCROLLKEEPER_DB_DIR) should be set to 
$(localstatedir)/lib/scrollkeeper.  If not, $(SCROLLKEEPER_DB_DIR) 
should be set to `scrollkeeper-config --pkglocalstatedir`.

The -p option is important for building and testing purposes and should
always be used when packaging applications.  When building RPMs, for
example, it guarantees that instead of actually modifying the primary
database on the builder's machine, it generates and modifies a new
database under BUILDROOT.  It is clearly useful for testing purposes
as well.

scrollkeeper-update identifies the presence of a new OMF file in 
OMFDIR and registers new metadata.

Uninstalling is done using 'scrollkeeper-update' as well.

uninstall:
	rm $(DOCDIR)/foo-manual.sgml 
	rm $(OMFDIR)/foo-manual-fr.omf
	scrollkeeper-update -p $(SCROLLKEEPER_DB_DIR)

scrollkeeper-update identifies that the OMF file was removed from
OMFDIR and unregisters the old metadata.

III. Other Tools for Admin and Packaging

In addition to the functions described above (scrollkeeper-preinstall
and scrollkeeper-update), ScrollKeeper provides the following additional
utilities:

scrollkeeper-rebuilddb:
This is used to rebuild the ScrollKeeper database completely using only
the original OMF files in OMFDIR.   It also has a "-p" option which
allows one to specify the output directory of the newly created 
database files.  

scrollkeeper-config:
This is used to return various configuration paths for ScrollKeeper. 
  Options:
    --help              Prints scrollkeeper-config usage.
    --version           Prints Scrollkeeper version.          
    --prefix            Scrollkeeper configure prefix. (eg. /usr)
    --localstatedir     Scrollkeeper configure localstatedir. (eg. /var)
    --pkglocalstatedir  Scrollkeeper data directory. (eg. /var/lib/scrollkeeper)
    --pkgdatadir        Scrollkeeper home directory. (eg. /usr/share/scrollkeeper)
    --omfdir		Directory where ScrollKeeper looks for OMF files.
			(eg. /usr/doc/omf)

IV. ScrollKeeper Internals

This section describes some details about scrollkeeper functions.

$SCROLLKEEPER_DB_DIR is the directory where ScrollKeeper keeps its data
files.  (It is typically under /var.)

$SCROLLKEEPER_DIR is the directory where ScrollKeeper installs its
static non-database files, such as the README and license files.  (It is 
typically under /usr or /share.)

$OMFDIR is the directory where packages must install their OMF files and
where ScrollKeeper looks to find OMF files. (This is typically 
$prefix/doc/omf.)

scrollkeeper-update -p <scrollkeeper_db_dir>:
  (Note: the -p flag is optional.  If omitted, the path is determined
   using `scrollkeeper-config --pkglocalstatedir`. However this flag
   should not be omitted when scrollkeeper-update is used in Makefiles
   since building often uses BUILDROOT's.)
  It looks for added, removed, or updated files in OMFDIR and then calls 
  scrollkeeper-install and scrollkeeper-uninstall as necessary.

scrollkeeper-install -p <scrollkeeper_db_dir> <omfname>
  1) Validate <omfname> 
  2) For each entry in <omfname> do the following:
    A) Verify that the URL in the OMF entry exists.  (eg. the file
       it points to on the drive actually exists.) If it fails this
       test, abort all further action on this OMF entry.
    B) Add the entry to <scrollkeeper_db_dir>/scrollkeeper-docs.
       (This file is created if it doesn't exist.) (see V. for format)
    C) Update Contents List:
      a) Determine title, language, and position(s) in Contents List from
         TITLE, LANGUAGE, and SUBJECT.toc respectively.  
      b) Verify that the SUBJECT.toc lies within our controlled list
         specified by `scrollkeeper-config --pkgdatadir`/contents-tree.xml  
         (Do we really want ScrollKeeper to rigorously enforce the 
         controlled list? Perhaps this should be left up to the help browser?)
      c) Add this document to the Content List file in the appropriate 
         language's directory: 
         <scrollkeeper_db_dir>/<LANGUAGE>/scrollkeeper-contents-list.xml  
         (See IV. for format.) (Notice the doc ID is used as an attribute
         to make it uninstallation easy.)
         (This file is created if it doesn't exist.)
    D) (for SGML and XML docs only) Extract TOC from doc, placing it 
       into a file: <scrollkeeper_db_dir>/TOC/<IDNUMBER>  
       (format to be determined)
       (This file is created if it doesn't exist, or overwritten if it does.)
    E) (for SGML and XML docs only) Insert TOC into an 
       "extended contents list".  This is the Contents List with TOC listed 
       as a sub-tree for each doc.  As above, the doc ID is used to make 
       uninstallation easy.  This file is at 
       <scrollkeeper_db_dir>/<LANGUAGE>/scrollkeeper-extended-contents-list.xml
       (This file is created if it doesn't exist.)

scrollkeeper-uninstall -p <scrollkeeper_db_dir> <omfname>
    [ Fill in later. Should be trivial since the contents-lists and
      toc are all identified by doc id.]

scrollkeeper-rebuilddb -p <scrollkeeper_db_dir>:
  (Note: The option '-p' is optional.  If omitted, the path is determined
   using `scrollkeeper-config --pkglocalstatedir`.)
  This will cause scrollkeeper to:
    a) Move all of its files under <scrollkeeper_db_dir> into some 
       temporary location.  (or maybe just delete them.) 
    b) Run scrollkeeper-update -p <scrollkeeper_db_dir> to enter
       all OMF files into a new database in <scrollkeeper_db_dir>.
  This can be used whenever the database is corrupted or becomes out-of-
  date.  It is also an easy (brute-force) way to allow for scrollkeeper 
  to update its databases when a new version is installed.

-- For use by help browsers --
scrollkeeper-get-contents-list <language>:
  1) Returns the path:
        ${pkglocalstatedir}/<language>/scrollkeeper-contents-list.xml

scrollkeeper-get-extended-contents-list <language>:
  1) Returns the path:
        ${pkglocalstatedir}/<language>/scrollkeeper-extended-contents-list.xml

scrollkeeper-get-toc-from-docpath <docpath>:
  1) Looks in ${pkglocalstatedir}/scrollkeeper-docs to find doc id for <docpath>
  2) Returns the path:
        ${pkglocalstatedir}/TOC/<docid>

[ Note that extending this all to include the Index should be very
  similar. ]

V. Example File Snippits 

scrollkeeper-docs:
	This is used to keep track of which docs are on the system 
	(and possibly some remote docs), listing the location of
	their OMF metadata, and identifying each doc by a code which
	can be used to simplify uninstallation and some other tasks.
	It also has a timestamp from when the OMF file was last written
	so that scrollkeeper-update can detect when an OMF file has
	been updated.    

/usr/doc/omf/foo-manual-fr.omf     000001  /usr/doc/foo/foo-manual.sgml   Sun Oct 29 13:48:50 
/usr/doc/omf/bar-manual-en_US.omf  000002  /usr/doc/bar/bar-manual.sgml   Tue Oct 24 02:47:28
/usr/doc/omf/foob-manual-da.omf	   000003  /usr/doc/foob/foob-manual.sgml Tue Oct 24 02:47:28

contents-tree.xml:
	This is the controlled list for valid Contents List sections.

scrollkeeper-contents-list.xml:
	This is a simple XML file which reveals what the Contents List
	looks like and where the OMF files and doc are at. This is used
	by the help browser to display the Contents List.
	[We will need to write up a simple DTD.]

   <sect>
     <title>Applications</title>
     <sect>
       <title>Games</title>
       <doc docid="000050">
         <doctitle>FreeCell</doctitle>
         <docomf>/usr/local/games/freecell/freecell-omf.xml</docomf>
         <docsource>/usr/local/games/freecell/freecell.sgml</docsource>
         <docformat>SGML</docformat>
       </doc>
     </sect>
   </sect>

TOC/<IDNUMBER>:
	Contains a description of the sectioning layout of the doc.
	It also must provide URI's for each doc section.  If every
	section has an "id", then this is straightforward.  If not,
	then .... (?? FILL THIS IN ??)  (How should we store this?
	XML or a flat file?)

scrollkeeper-extended-contents-list.xml:
	Similar to scrollkeeper-contents-list.xml, except has TOC info in
	it too.  This is used by the help browser to display the
	"extended Contents List".

   <sect>
     <title>Applications</title>
     <sect>
       <title>Games</title>
       <doc docid="000050">
         <doctitle>FreeCell</doctitle>
         <docomf>/usr/local/games/freecell/freecell-omf.xml</docomf>
         <docsource>/usr/local/games/freecell/freecell.sgml</docsource>
         <docformat>SGML</docformat>
         <toc>
          <tocsect1 linkid="introduction">Introduction to FreeCell</tocsect1>
          <tocsect1 linkid="playing">Playing FreeCell
           <tocsect2 linkid="winning">Winning FreeCell</tocsect2>
          </tocsect1>
         </toc>
       </doc>
     </sect>
   </sect>

foo-manual.omf:
  [FIXME]

Thread: [Scrollkeeper-devel] ScrollKeeper-0.1 Proposal #3

scrollkeeper-devel