[Docutils-develop] Re: [Doc-SIG] looking for prior art

[Doug H. <do...@he...>]

On Thursday 05 December 2002 9:45 pm, David Goodger wrote:
> Doug Hellmann wrote:
> > I'm pretty sure HappyDoc was written before the compiler module was
> > generally available
>
> I suspected as much.  Either that, or you're a glutton for punishment
> ;-)

Well, I didn't say that wasn't true.  :-)  I actually started with some 
sample code included in the Python source distribution, so it wasn't too hard 
to extend it and come up with a useful parser.

> > I've only had to make a few minor modifications to it in the past,
> > since the language syntax hasn't evolved that far.
>
> That's good to know.  Still, the parser.suite() approach seems a lot
> harder.

If you're starting from scratch, I would definitely recommend trying the 
compiler module first.

> > I'm working on a major overhaul of HappyDoc anyway, so now might be
> > the time to rewrite the parsing stuff to use the compiler module.
> > If you're interested in collaborating, let me know.
>
> I am, definitely.  What I'd like to do is to take a module, read in
> the text, run it through the module parser (using compiler.py and
> tokenize.py) and produce a high-level AST full of nodes that are
> interesting from an auto-documentation standpoint.  For example, given
> this module (x.py)::

[...]

> compiler.parse() provides most of what's needed for this AST.  I think
> that "tokenize" can be used to get the rest, and all that's left is to
> hunker down and figure out how.  We can determine the line number from
> the compiler.parse() AST, and a get_rhs(lineno) method would provide
> the rest.

Does compiler include comments?  I had to write a separate parser to pull 
comments out.

> The Docutils Python reader component will transform this AST into a
> Python-specific doctree, and then a `stylist transform`_ would further
> transform it into a generic doctree.  Namespaces will have to be
> compiled for each of the scopes, but I'm not certain at what stage of
> processing.

Why perform all of those transformations?  Why not go from the AST to a 
generic doctree?  Or, even from the AST to the final output?

> It's very important to keep all docstring processing out of this, so
> that it's a completely generic and not tool-specific.

Definitely.

Doug