Re: [Docutils-users] Programming reference documentation

[David G. <go...@py...>]

I'm copying this to docutils-develop, a more appropriate venue for the
discussion IMO.  Subscribe at
<http://lists.sf.net/lists/listinfo/docutils-develop>.

Ian Bicking wrote (quoted in full):
> I'm working with the Webware project (webware.sf.net), and we really
> need some cohesive reference documentation, aimed particularly at
> the public interfaces.  I'm wondering if anyone is using docutils
> for this now... if so, I'd like to see how it's working.  It's not
> entirely clear to me what docutils status is (I've used reST a lot,
> but not much else).

Are you referring to "auto-documentation" (extracting documentation
from source code)?  I have begun work on a Python Source Reader
component which will accomplish this.  The code so far is in
docutils/readers/python/moduleparser.py.  The plan is described in PEP
258 (<http://docutils.sf.net/spec/pep-0258.html>),
<http://docutils.sf.net/spec/pysource.html>, and
<http://docutils.sf.net/spec/notes.html#python-source-reader>.

> Some options I've considered, in order of complexity:
> * Writing the whole thing in reST, separate from the code itself.
>   The writing wouldn't be so bad, but the maintenance would be more
>   difficult.

This can be done now of course.

> * Splitting it up into modules, and putting the entire reference
>   documentation in the module docstring.

The Python Reader package doesn't extract docstrings yet.  I'm
currently concentrating on finding a new job [#]_, which means I don't
know how much time I'll be able to devote to this.

.. [#] Check out my resume at
   <http://starship.python.net/~dgoodger/cv/>!  Please tell your
   friends, relatives, neighbors, managers, and HR people!  Let me
   know if you have any leads.  Thanks!

> * Using internal references in those docstrings, so I could
>   effectively include the docstrings from other portions of code
>   (classes, methods, etc).

I hadn't thought of that approach.  From your description, it seems to
require a lot of bookkeeping and redundancy, which I wouldn't want to
write or require others to write.  But I encourage you to explore it.

> One thing I *don't* like is reference documentation in the style of
> pydoc.  To me it's too automatic, and generally too hard to follow.

I hope that Docutils' auto-documentation will be more flexible than
pydoc's, although it will have the same basis.  It will certainly have
more flexible rendering, and I hope much more readable.

> I find reference documentation with some framework of text (lead-in
> paragraphs, categorization, etc) to be much better.

That can be tricky.  One idea I had (documented in the spec/notes
file) is to recognize JavaDoc-style documentation comments almost
anywhere in the code.  This would allow a more "literate programming"
approach than what pydoc provides.

> There are also
> methods which, while "public" aren't actually useful, and I don't
> think those should be documented -- they are just distracting.

How could software tell the difference?  If we can come up with an
unobtrusive way to tell the docstring processing system to disregard
certain methods, it could be useful.  Maybe a class attribute
equivalent of the "__all__" module attribute?  Or perhaps the
opposite: a "__none__" class attribute ;).

> So the last option is the one I like the most.  But I doubt it's in
> place.

Correct, it's not.

> I've already decided I'm not going to get the reference
> documentation done for the next release (there's other documentation
> that needs to be written, and documentation is tiring :), so I don't
> plan to use this in the short term and I could put some work into
> producing a system I like.

Glad for the help!

> My vision for a docstring-based reference documentation would
> probably look like:
> 
> """
> Module
> ======
> 
> This module does X, Y, Z.  Most work will be done through the class:
> 
> .. docstring:: MainClass
> 
> In certain cases you may wish to access the functionality
> separately, yada yada yada, and may wish to use:
> 
> .. docstring:: some_function
> .. docstring:: some_other_function
> """
> 
> class MainClass:
>     """
>     Recursively we continue, now with methods:
> 
>     .. docstring:: MainClass.some_method
>     """

I envision something more along the lines of this::

    """
    A general description of the highlights of the module, with
    references (interpreted text) to the details.  We don't need
    a "Module" title because it will be derived automatically.
    
    This module does X, Y, Z.  Most work will be done through the
    `MainClass` class.
    
    In certain cases you may wish to access the functionality
    separately, yada yada yada, and may wish to use `some_function`
    or `some_other_function`
    """
    
    class MainClass:
        """
        Description of this class, with references to specific
        methods like `some_method`.  We don't need to fully qualify
        (no "MainClass." prefix) because "some_method" is in the
        local namespace.
        """

        ##
        # My idea from JavaDoc is to allow document comments
        # interspersed in the code in some way, which will
        # interleave with the code objects and docstrings.
        # These would include section titles, allowing a
        # subdivision of methods by category.  For example:
        
        ##
        # Magic Methods
        # =============
        
        def __str__(self):
            """String representation of this object (pseudo-XML)."""
            ...

> How far away might this be?

A function of my spare time and volunteer contributions.  Hint hint.

> Are there better alternatives already in place, or planned on?

Alternative software packages?  There's HappyDoc, EpyDoc, effbot's
PythonDoc, and lots of older stuff too; see PEP 256.  None use
reStructuredText though.

> Does someone experienced in writing
> reference documentation think this is the wrong way to go about it?

ISTM that there's a built-in limit to the potential quality of
auto-documentation.  It may not lend itself to a more narrative or
free-form style.  However, I hope that Docutils will be a platform for
experimentation.  That's why I'm keeping the processing stages
separate and well-defined.  See
<http://article.gmane.org/gmane.text.docutils.devel/139> and the
thread starting with
<http://article.gmane.org/gmane.text.docutils.devel/118>.

-- 
David Goodger    http://starship.python.net/~goodger    Projects:
  * Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  * The Go Tools Project: http://gotools.sourceforge.net/