Re: [Docutils-develop] Custom node types (was Re: pdf writer)

[David G. <go...@us...>]

Tony J Ibbs (Tibs) wrote (about "DPS nodes"):
> (what *is* the current correct term for these entities, by the way,
> now that DPS has gone away?)

I've been calling them "Docutils nodes".  They come from the
"docutils.nodes" module after all.

[David]
> > In the case of PySource, an example of (a), I would expect to have
> > a transform reduce any custom nodes to standard node structures...

[Tibs]
> Ah. The way pysource works is to build an internal structure
> representing the Python code, insofar as it cares (i.e., it doesn't
> try to represent stuff it's not interested in), and then transform
> *this* into DPS nodes.

That's fine.  I'd just be wary of converting to standard Docutils
nodes too early, because then you're locking in one style.  This comes
back to a discussion we had some time ago, about "stylist"
components...

> This means that (within pysource) I have no need of "custom" nodes -
> the DPS datastructures are purely being built for consumption by a
> Writer

I think that a set of custom nodes in conjunction with some "stylist"
transforms would work well.  The PySource Reader wouldn't have to make
any decisions as to style; just produce a logical tree, parsed &
linked.  Stylist transforms would understand the custom nodes; one of
them would turn the custom nodes into a standard Docutils nodes.

The point is that by decoupling the context-understanding part of the
Reader from the layout-generating part(s), the whole becomes much more
flexible.  I'm not a big fan of the PyDoc-style layout, and PySource
is similar.  I'd like to be able to experiment with radically
alternate layouts without having to touch the AST-grokking part of the
code.  To do that, I need access to the parsed data at an early stage,
before it's been altered too much (at all) stylistically.

PySource could have a "--stylist" option specifying which stylist
transform is desired.  In between the Reader proper and the transform,
custom nodes would be fine, and probably preferable to standard
Docutils nodes.

> (it's obviously a little bit more complex than that, because
> docstrings become DPS fragment trees within my "python"
> datastructure, but that makes sense in terms of treating individual
> docstrings as entire documents for the purposes of footnotes,
> headers, etc.).

This is where a "merge" transform would come in handy.  See
http://docutils.sf.net/spec/notes.html#reference-merging.

To get all the details right there's lots of work to do!

> As I indicated above, I don't store Python structure "as such" in
> the DPS node tree - I just store the document that "talks about" the
> Python structure.

They're close parallels though, aren't they?  By storing the 'document
that "talks about" the Python structure' too soon, you lose the
flexibility to render that document in multiple ways.

> Since the pysource.dtd is talking about DPS nodes, I'm not
> particularly interested (after all, they're not nodes a Writer is
> required to recognise).

But if the "internal structure representing the Python code" was made
up of Docutils nodes (custom & standard), you could make use of the
full Docutils transform machinery, without having to roll your own.

I need to work out and explain the whole inner workings better.  I
*know* that if you understood the Docutils model better, you'd realize
the benefits of the approach I'm advocating.  Not enough hours in the
day, and not enough days in the weekend.  Gradually we'll get there;
this is part of the process.

> The reason I want "group" is that one of the things I want to be
> able to do is to group together, visually, for instance, a section
> heading and some text thereafter (the layout is wrong, to get the
> idea across in ASCII art)::
> 
>     Class Fred                          <-- a "section heading"
>     fullname: jimbob.clancy.Fred        <-- a paragraph or somesuch
>     subclasses: BigClass                <-- ditto

But "group" has no intrinsic meaning; it's too generic.  I'd rather
see a whole bunch of specialized "groups", each specific to its task.
The PySource DTD is a list of such groups.

Incorporating your ideas (good ones, too), I've revised
http://docutils.sf.net/spec/pysource.dtd quite a bit.  Here's the new
"class_section" and "fullname" element specs::

    <!ELEMENT class_section
        (class, inheritance_list?, fullname?, subclasses?,
         %structure.model;)>
    <!ELEMENT fullname (package | module | class | method | function)+>

So a class_section begins with a class name, then an inheritance list,
then a full name, and a list of subclasses.  A "fullname" is a list of
references, rendered as a dotted name.  "inheritance_list" and
"subclasses" are lists of class references.  In HTML, each of the
references could be clicked on, providing navigation.

If the internal document tree used such nodes, it would be easy to
transform them into a table-based colorful structure like now, or a
more austere linear style, or others I can't even imagine now.  It
looks to me like you want to render "fullname" and "subclasses" as a
field list; a stylist could do so.

Saying that you "want to be able to ... group together, visually"
something is a dead giveaway.  PySource is a Reader, and should not be
concerned with style *at all*.  Writers shouldn't be concerned with
input context either, so the Python-specific stuff has to be gone by
the time the document gets that far.  An intermediate stylist
transform is the answer.

If the piece of code that does the styling is kept small and modular,
it would be much easier for people to roll their own styles.  The
"barrier to entry" is too high now; extracting the stylist code would
lower it considerably.

You must resist the urge to mix content and style.  That way lies the
dark side.  If once you start down the dark path, forever will it
dominate your destiny, consume you it will.  Keeping them separate is
more work initially, but pays off enormously in the end.  We're
already starting to see the payoff with Docutils, with PDF and
MoinMoin Writers under development.

> but not the text that follows *that* - which might, for instance, be a
> docstring. Now, I may be able to do that with CSS just by use of the
> "class" values, but I had thought not

CSS is limited.  CSS1, which is well-supported in many of today's
browsers, can't do much more than decorate; it can't transform
structures the way you want.  CSS2 can do more, but it's not supported
well yet.

> And I'm pretty sure I could do the same sort of thing in TeX (it's
> really the TeX concept of "group" - i.e., ``{\something .. }`` -
> that I'm after, I think!).

Docutils nodes (described by the Docutils DTD) are exactly analogous
to TeX structural markup.

> It may be that my aims are evil, but they are certainly simply and
> easily met by provision of the equivalent of a "group" node (which may,
> of course, be translated into a table in a non-CSS environment).

I want to give you a *plethora* of group nodes; one just won't cut it!

Enjoy the sun!

-- 
David Goodger  <go...@us...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/