docutils-develop

[Docutils-develop] Re: sandbox/tibs/pysource/notes thoughts.txt

[Richard J. <rj...@ek...>]

On Sat, 12 Oct 2002 8:45 am, David Goodger wrote:
> I reckon that one produces a Python-specific doctree, and then
> chooses one of several transforms to produce generic doctrees.

A sound plan.


> By the way, I still vote for "static" introspection

+1


> One of the tasks undertaken by the Python Source Reader is to
> decide what to do with docstrings.  For each Python file, it
> discovers (from ``__docformat__``) if the input parser is
> "reStructuredText". If it is, then the contents of each docstring
> in that file will be parsed as such, including sorting out
> interpreted text (i.e., rendering it into links across the tree).
> If it is not, then each docstring will be treated as a literal
> block.
>
>   This admits the posibility of adding other parsers *for
>   docstrings* at a later date - which I suspect is how HappyDoc
>   does it. It is just necessary to publicise the API for the
>   Docstring class, and then someone else can provide alternate
>   plugins.
>
> The output of the Python Source Reader is thus a Python-specific
> doctree, with all docstrings fully processed (as appropriate),
> and held inside <docstring> elements.
>
> The next stage is handled by the Layout Transformer.
>
>     [What I call "stylist transforms".  --DG]
>
> This determines the layout of the document to be produced - the
> *kind* or *style* of output that the user wants (e.g., PyDoc
> style, LibRefMan style, generic docutils style, etc.).
>
> Each layout is represented by a class that walks the
> Python-specific doctree, replacing the Python-specific nodes with
> appropriate generic nodes. The output of the Layout Transformer
> is thus a generic doctree.

It's going to have to be many generic doctrees though. That is, the "stylist 
transforms" are likely to produce multiple output doctrees (an index, a 
package page, multiple module pages, ...)

The resultant docutils doctrees are passed via normal transforms to the normal 
HTML writer.

I can see the multiple page output being useful for documenting larger volumes 
of text - it'd be really nice if my Roundup documentation could all be parsed 
in one go and then I could have inter-document references verified (and point 
to specific anchors, oh my :), have the top-level Table Of Contents for all 
documents generated, ...


> One specific thing to be decided, particularly for HTML, is
> whether one is outputting a "cluster" of files (e.g., as javadoc
> does).

Haven't looked at javadoc for a while - what's this mean?


    Richard

Re: [Docutils-develop] Re: sandbox/tibs/pysource/notes thoughts.txt

[David G. <go...@us...>]

> On Sat, 12 Oct 2002 8:45 am, David Goodger wrote:

Actually, Tibs wrote it.  I just checked it in to CVS.

Read it here: 
http://docutils.sf.net/sandbox/tibs/pysource/notes/thoughts.html

>> Each layout is represented by a class that walks the
>> Python-specific doctree, replacing the Python-specific nodes with
>> appropriate generic nodes. The output of the Layout Transformer
>> is thus a generic doctree.

Richard Jones wrote:
> It's going to have to be many generic doctrees though. That is, the "stylist
> transforms" are likely to produce multiple output doctrees (an index, a
> package page, multiple module pages, ...)

I don't know how this will work yet.  Perhaps the Writer and/or I/O object
will handle forests (multiple trees).  Perhaps the doctree will remain
monolithic, and a transform will insert "split here" indicators.  It's all
up in the air.

>> One specific thing to be decided, particularly for HTML, is
>> whether one is outputting a "cluster" of files (e.g., as javadoc
>> does).
> 
> Haven't looked at javadoc for a while - what's this mean?

I think "cluster" here just means multiple files as opposed to one
monolithic file.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] Re: sandbox/tibs/pysource/notes thoughts.txt

[Richard J. <rj...@ek...>]

On Sat, 12 Oct 2002 10:37 am, David Goodger wrote:
> > On Sat, 12 Oct 2002 8:45 am, David Goodger wrote:
>
> Actually, Tibs wrote it.  I just checked it in to CVS.

Sorry, slackness on my part there.


> >> Each layout is represented by a class that walks the
> >> Python-specific doctree, replacing the Python-specific nodes with
> >> appropriate generic nodes. The output of the Layout Transformer
> >> is thus a generic doctree.
>
> Richard Jones wrote:
> > It's going to have to be many generic doctrees though. That is, the
> > "stylist transforms" are likely to produce multiple output doctrees (an
> > index, a package page, multiple module pages, ...)
>
> I don't know how this will work yet.  Perhaps the Writer and/or I/O object
> will handle forests (multiple trees).  Perhaps the doctree will remain
> monolithic, and a transform will insert "split here" indicators.  It's all
> up in the air.

The "split here" part worries me - that then implies the HTML Writer knows how 
to output multiple files, which I don't believe it should. It should be the 
job of the pysource "framework" (the bit that drives the parse -> transform 
-> writer mechanisms) to decide what files need to be written and how they 
should be written.


> >> One specific thing to be decided, particularly for HTML, is
> >> whether one is outputting a "cluster" of files (e.g., as javadoc
> >> does).
> >
> > Haven't looked at javadoc for a while - what's this mean?
>
> I think "cluster" here just means multiple files as opposed to one
> monolithic file.

Then this is definitely an issue that must be resolved now ;)


   Richard

Re: [Docutils-develop] Re: sandbox/tibs/pysource/notes thoughts.txt

[David G. <go...@us...>]

> On Sat, 12 Oct 2002 10:37 am, David Goodger wrote:
>> I don't know how this will work yet.  Perhaps the Writer and/or I/O object
>> will handle forests (multiple trees).  Perhaps the doctree will remain
>> monolithic, and a transform will insert "split here" indicators.  It's all
>> up in the air.

Richard Jones wrote:
> The "split here" part worries me - that then implies the HTML Writer knows how
> to output multiple files, which I don't believe it should. It should be the
> job of the pysource "framework" (the bit that drives the parse -> transform
> -> writer mechanisms) to decide what files need to be written and how they
> should be written.

Problem is, PySource is a *Reader*, which doesn't know and shouldn't care
about final output.  HappyDoc may have already tackled this issue; analyzing
its code is on my to-do.  But also on my to-do are window trim, painting,
grouting, baseboard mouldings, winterizing the garage, and on and on...

> Then this is definitely an issue that must be resolved now ;)

I hope you're volunteering :-)

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

Re: [Docutils-develop] Re: sandbox/tibs/pysource/notes thoughts.txt

[Richard J. <rj...@ek...>]

On Sat, 12 Oct 2002 11:01 am, David Goodger wrote:
> > On Sat, 12 Oct 2002 10:37 am, David Goodger wrote:
> >> I don't know how this will work yet.  Perhaps the Writer and/or I/O
> >> object will handle forests (multiple trees).  Perhaps the doctree will
> >> remain monolithic, and a transform will insert "split here" indicators. 
> >> It's all up in the air.
>
> Richard Jones wrote:
> > The "split here" part worries me - that then implies the HTML Writer
> > knows how to output multiple files, which I don't believe it should. It
> > should be the job of the pysource "framework" (the bit that drives the
> > parse -> transform -> writer mechanisms) to decide what files need to be
> > written and how they should be written.
>
> Problem is, PySource is a *Reader*, which doesn't know and shouldn't care
> about final output.  HappyDoc may have already tackled this issue;
> analyzing its code is on my to-do.  But also on my to-do are window trim,
> painting, grouting, baseboard mouldings, winterizing the garage, and on and
> on...

I don't think pysource is going to be able to get away with being *just* a 
Reader. The one-source-to-many issue kinda throws a huge spanner in the 
works.


> > Then this is definitely an issue that must be resolved now ;)
>
> I hope you're volunteering :-)

Heh, sure. It can go on my TODO along with Roundup 0.5.1, weeding and 
generally attacking the garden, finish the bedside tables, hallstand, 
bookcases, checking the roof after our recent strong winds, ... :)


   Richard

[Docutils-develop] split output (was Re: sandbox/tibs/pysource/notes thoughts.txt)

[David G. <go...@us...>]

[David]
>>>> I don't know how this will work yet.  Perhaps the Writer and/or I/O
>>>> object will handle forests (multiple trees).  Perhaps the doctree will
>>>> remain monolithic, and a transform will insert "split here" indicators.
>>>> It's all up in the air.

[Richard]
>>> The "split here" part worries me - that then implies the HTML Writer
>>> knows how to output multiple files, which I don't believe it should. It
>>> should be the job of the pysource "framework" (the bit that drives the
>>> parse -> transform -> writer mechanisms) to decide what files need to be
>>> written and how they should be written.

[David]
>> Problem is, PySource is a *Reader*, which doesn't know and shouldn't care
>> about final output.

[Richard]
> I don't think pysource is going to be able to get away with being *just* a
> Reader. The one-source-to-many issue kinda throws a huge spanner in the
> works.

The one-source-to-many issue is pervasive, not specific to PySource at all.
For example, I'd like to be able to split the "reStructuredText Markup
Specification" into multiple .html files, one per first-level section (and
perhaps even one per second-level section).  "The Docutils Document Tree: A
Guide to the Docutils DTD", whose HTML is now up to 208K, is becoming much
too big.  And it's far from finished!

I agree that the HTML Writer shouldn't have to know how to split & write
multiple files.  This may be a job for a specialized I/O object, like
"MultiFileOutput".  It will need some earlier support though, such as
reference adjustment (any given reference may need to point to
``href="file#id"`` instead of just ``href="#id"``).  Perhaps I/O objects
should become full-fledged components (i.e. subclasses of
``docutils.Component``, as are Readers, Parsers, and Writers now), and thus
have associated option/setting specs and transforms.

In any case, we should try to find an elegant, general solution to the
problem.  We may not get there with the first attempt, or even the second,
but that's OK.  One thing this project has taught me is to let go of code,
to be willing to sacrifice it when it has proven unfit.

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/