RE: [Docutils-develop] split output (was Re: sandbox/tibs/pysource/notes thoughts.txt)

[Tony J I. (Tibs) <to...@ls...>]

Gosh - I come in Monday morning and there's email to reply to...
I'll try to amalgamate several replies in this thread...

Early on, Richard Jones wrote:
>> One specific thing to be decided, particularly for HTML, is
>> whether one is outputting a "cluster" of files (e.g., as javadoc
>> does).
>
> Haven't looked at javadoc for a while - what's this mean?

As David guessed, it means the sort of thing that javadoc does -
specifically, it produces a directory structure, containing (in general)
an HTML file for each class, a directory for each package, index and
contents pages, maybe a summary page for each package, and (I think)
other bumf as well. One single Java source file can, all by itself,
produce a whole set of files.

  ( So, Richard, is that resolved now? :)

Now, sometimes I want all of my documentation in one document (it's a
bugbear of mine about the current Python development docs that you only
get this form published when they're released, not when they're being
developed - chapters in lots of little bits, yuck), and sometimes I want
it split up for fast loading, etc. I imagine we *will* want to provide
these options for pysource.

On the other hand, this seems to me like a minor issue compared to
getting the *information* right - that is, sorting out the data mining,
the joining up of `interpreted-Python` links, etc.

On the other other hand, early adoption probably *does* depend on
getting nice presentation. Life's fun that way.

Richard Jones wrote:
> It's going to have to be many generic doctrees though. That is, the
"stylist
> transforms" are likely to produce multiple output doctrees (an index,
a
> package page, multiple module pages, ...)

David Goodger replied:
> I don't know how this will work yet.  Perhaps the Writer and/or I/O
object
> will handle forests (multiple trees).  Perhaps the doctree will remain
> monolithic, and a transform will insert "split here" indicators.  It's
all
> up in the air.

I can certainly see other document-producers wanting to split output
(the obvious case being people doing "book" production, where individual
chapter output can be quite useful). I suspect, however, that pysource
would be a good environment to test out ideas, since it probably has the
most extreme requirements (or perhaps that should be "wish list").

Richard Jones then wrote:
> The "split here" part worries me - that then implies the HTML Writer
knows how
> to output multiple files, which I don't believe it should. It should
be the
> job of the pysource "framework" (the bit that drives the parse ->
transform
> -> writer mechanisms) to decide what files need to be written and how
they
> should be written.

to which David responded:
> Problem is, PySource is a *Reader*, which doesn't know and shouldn't
care
> about final output.

Well, no, pysource is *either* a module providing facilities callable by
other software, *or* the default application using said module. And in
either case, it will be at least a Reader and a set of transformers for
the output of said reader (after all, it has to transform stuff from its
"special" docutils tree into the generic tree, and there will be options
on how to do that).

Anyway, if, as Richard wants (and I had assumed) the default way of
working is to read in "everything we care about" (gosh, memory) so that
one can get overview contents and index pages, and then output it as
single or multiple documents, then I think that David is right - one of
the transform phases needs to identify where to split to produce a
forest from the original tree (since we're assuming a standard Writer,
we need to split things up into separate document trees before it gets
fed things - i.e., it's easiest to imagine that a Writer only writes one
file, at least for the moment).

That splitting can either be on explicit break-points (which makes for a
very simple splitting utility), or might be on designated locations -
e.g., for each class. That latter would assume that more "fix up" work
would be needed.

In short, I see the process as something like (and my terminology is
going to be all astray again - sorry::

    PySourceReader
      :produces:
    <PySource doctree>
      :through:
    PySourceTransformer
      :produces:
    <Standard doctree>
      :through:
     StandardWriter

*or*::

                PySourceReader
                  :produces:
               <PySource doctree>
                  :through:
               PySourceTransformer
                  :produces:
    <Standard doctree1> <Standard doctree2> ...
      :through:           :through:
     StandardWriter      StandardWriter     ...

If/when we get a "generic" split transform, we might shift that to::

                PySourceReader
                  :produces:
               <PySource doctree>
                  :through:
               PySourceTransformer
                  :produces:
              <Standard doctree1>
                  :through:
              <SplitTransformer>
                  :produces:
     <Standard doctreeA>  <Standard doctreeB>
      :through:            :through:
     StandardWriter       StandardWriter     ...

However, whilst this is all fun, I think it's the technically easy bit
(the *design* of this is the hard bit), and for pysource itself, output
is still the least of our worries (having said that, this is all
volunteer work, and if someone thinks they'd rather fix this lacuna
first, then that's their business).

Tibs, who must stop wittering and go back to work.

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
.. "equal" really means "in some sense the same, but maybe not
.. the sense you were hoping for", or, more succinctly, "is
.. confused with". (Gordon McMillan, Python list, Apr 1998)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)