Docutils: Documentation Utilities

On Mon, 2002-10-28 at 21:40, David Goodger wrote:
> > To serve as an instructional sample I'm making a very simply Wiki --
> > I'd like to use reST as the markup, since it's not an example in
> > parsing (and I like the reST markup more than most Wiki markups to
> > boot).
> 
> Great!  Please share the results if you can.  I know that some
> intregration with MoinMoin has been done, although incomplete I
> believe.  See
> http://twistedmatrix.com/users/jh.twistd/moin/moin.cgi/RestSample

I will, though I'm not really making a full Wiki -- I just thought it
was a good example web application that's neat, yet potentially very
simple (especially when you don't make your own parser).

> Perhaps the tools/pep2html.py module is a better example, function
> "fix_rst_pep".  Docutils is called in one (logical) line of code::
> 
>     output = core.publish_string(
>         source=''.join(input_lines),
>         source_path=inpath,
>         destination_path=outfile.name,
>         reader_name='pep',
>         parser_name='restructuredtext',
>         writer_name='pep_html',
>         settings=docutils_settings)

Thanks, that's just what I wanted.

> > Also -- is there a good hook in reST that I can use for Wiki links?
> > The linking mechanism reST has is fine for internal and external
> > links, but doesn't fit right for Wiki links (which fall somewhere in
> > between).  If I could capture failed internal links and turn them
> > into inter-wiki-page links, that would be perfect.  Failing that, if
> > there was some way I could introduce another markup easily that
> > would be great.
> 
> There's mention of Wikis as an example potential Reader component in
> PEP 258 (http://docutils.sf.net/spec/pep-0258.html#readers).  A Wiki
> Reader could subclass the standard parser, or we could build in hooks
> if they prove general enough.  The PEP Reader
> (docutils/readers/pep.py) first subclassed the parser (still does,
> minimally: the "Inliner" class), but most of the code was moved into
> the parser proper for other client code to use.
> 
> I would suggest that Wiki links retain the trailing "_" of regular
> reStructuredText links, and a mechanism be added to do some kind of
> lookup from a Wiki target database.  The extra syntax (trailing "_")
> removes ambiguity inherent in CamelCase WikiLinks (such as anything
> written by Angus MacDuff).

That's definitely what I'd like to do -- I have never like WikiNames
anyway -- false positives and mangled case limit the medium, IMHO. 
There isn't any database of Wiki targets -- it's important that you be
able to make a link to a page that does not yet exist.  I think it would
be fine if Wiki links were made if no internal link could be found
(though with all the automatically-generated targets, there could be
some clashes).  Otherwise I'd need some other punctuation -- like `wiki
link`^ or somesuch.  There's a fair amount of seldom-used punctuation
that could be used.

In the case of Wiki links being a failback when no internal link was
found, it seems like the easiest way to handle it would be some sort of
error handler... maybe you already have something like this.  As to
adding new punctuation-based markup, I have no idea what interface would
make sense, except to somehow subclass the parser.  If the parser made
it easy to add certain styles of markup (either trailing punctuation, or
enclosing punctuation), then subclassing doesn't seem like a big burden.

As I think about it, I'm leaning toward new punctuation.  Where should I
look for this?

  Ian

> As I think about it, I'm leaning toward new punctuation.  
> Where should I look for this?

Please, don't do the punctuation. Stick with the trailing ``_``. Trust
me on this one. :) 

Regards,
Garth.

On Tue, 2002-10-29 at 11:06, Garth Kidd wrote:
> > As I think about it, I'm leaning toward new punctuation.  
> > Where should I look for this?
> 
> Please, don't do the punctuation. Stick with the trailing ``_``. Trust
> me on this one. :) 

Why, what's your experience?  Too hard to do, or too messy when you're
through?  My only concern is that I won't be able to distinguish between
internal and external links if _ gets more overloaded.

  Ian

>>> As I think about it, I'm leaning toward new punctuation.
>>> Where should I look for this?
>> 
>> Please, don't do the punctuation. Stick with the trailing 
>> ``_``. Trust me on this one. :)
> 
> Why, what's your experience?  Too hard to do, or too messy 
> when you're through?  

Too hard to deal with the people who find the existing reST syntax hard
enough to deal with before you throw even more punctuation into the mix?
:) 

> My only concern is that I won't be able to distinguish between 
> internal and external links if _ gets more overloaded.

Not at all. It's up to the consumer of the document to resolve links. In
reST terminology, that's either a parser or a formatter or something
else -- I can never guess -- but it's not the *parser* that does it. 

If you can make a link resolve within a document, fantastic! If not,
resolve it to other documents in the Wiki. That precedence also lets
people break titled sections in a document (automatic target) into a
separate page on the Wiki, in which case all of the links in the
document will suddenly point to the new page when the title is removed
from the source document. 

The same is true of links in the text. Let's say a site you used to link
to has disappeared. Pull a copy out of the Internet Archive, save as a
Wiki page, and remove the ``.. _target:`` entry. Done. 

If you used different punctuation, someone would have to wade through
the document changing all of the punctuation to do these things, which
is overly painful. 

The one thing I *can't* do is help you figure out how to do this in
code, because I haven't spent much time with reST in ages (okay, ever),
but I reckon changing target resolution is going to be a whole lot
easier than adding YetMorePunctuation_. 

Regards,
Garth.

On Tue, 2002-10-29 at 18:24, Garth Kidd wrote:
> If you can make a link resolve within a document, fantastic! If not,
> resolve it to other documents in the Wiki. That precedence also lets
> people break titled sections in a document (automatic target) into a
> separate page on the Wiki, in which case all of the links in the
> document will suddenly point to the new page when the title is removed
> from the source document. 
> 
> The same is true of links in the text. Let's say a site you used to link
> to has disappeared. Pull a copy out of the Internet Archive, save as a
> Wiki page, and remove the ``.. _target:`` entry. Done. 

Those are good arguments, you've convinced me.  Now I just need to know
how to do it :)

  Ian

I agree with Garth that new markup is not the way to go here.  But I also
agree with Ian that "trust me" just ain't good enough ;-), so I'm glad some
good convincing arguments came up.

[Ian]
>> My only concern is that I won't be able to distinguish between
>> internal and external links if _ gets more overloaded.

[Garth]
> Not at all. It's up to the consumer of the document to resolve links. In
> reST terminology, that's either a parser or a formatter or something
> else -- I can never guess -- but it's not the *parser* that does it.

Transforms, actually.  Hyperlinks are resolved in the transforms in
docutils.transforms.references.  The
docutils.transforms.universal.FinalChecks transform checks for unresolved
references and signals errors.  A transform replacing or preceding this one
would be the place to do the Wiki-linking.

> The one thing I *can't* do is help you figure out how to do this in
> code, because I haven't spent much time with reST in ages (okay, ever),

Garth *did* do a very cool overhaul of the unit test framework, for which he
has our eternal gratitude.

-- 
David Goodger  <goodger@...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

On Tue, 2002-10-29 at 19:59, David Goodger wrote:
> Transforms, actually.  Hyperlinks are resolved in the transforms in
> docutils.transforms.references.  The
> docutils.transforms.universal.FinalChecks transform checks for unresolved
> references and signals errors.  A transform replacing or preceding this one
> would be the place to do the Wiki-linking.

Wow, it actually works.  That was easier than I thought it would be. 
Here's what I did:

from docutils import nodes
from docutils.readers import standalone
from docutils.transforms import Transform

class WikiLinkResolver(nodes.SparseNodeVisitor):

    def visit_reference(self, node):
        if node.resolved or not node.hasattr('refname'):
            return
        refname = node['refname']
        node.resolved = 1
        node['refuri'] = refname
        del node['refname']

class WikiLink(Transform):

    default_priority = 400

    def apply(self):
        visitor = WikiLinkResolver(self.document)
        self.document.walk(visitor)
        
class Reader(standalone.Reader):

    supported = standalone.Reader.supported + ('wiki',)

    default_transforms = standalone.Reader.default_transforms \
                         + (WikiLink,)



######################################
# Then to actually use it I had to do:

from docutils import readers
# WikiReST is the module I gave above, and it's located here:
sys.path.append('/usr/home/ianb/prog/WebwareWiki/Wiki')
readers._reader_aliases['wiki'] = 'WikiReST'

# Then I use 'wiki' for the reader_name in docutils.core.publish_string

######################################

Of course, it would be good if there was a better hook for adding a
reader like that.  For instance, a better use of __import__ in
docutils.readers.__init__ would have kept me from having to add
something to the path (I could have used
"readers._reader_aliases['wiki'] = 'Wiki.WikiReST'"), and of course it
would be better still if I could have simply passed in the class to an
argument to publish_string.  Of course, now that I look at it, there's a
reader argument to publish_string -- but I have to give it an instance,
and that seem awkward to make -- how can I instantiate the reader if I
have to pass the parser to the constructor, and the parser is created in
publish_string?

I'm sure there's a better way, or you can add a better way ;)  But it
works without heroic efforts, and that's very cool.

Oh, and what exactly is the default_priority bit?  If I set it high,
will Wiki links be created before internal links (hence subsuming
internal links)?

  Ian

Ian Bicking wrote:
> Wow, it actually works.  That was easier than I thought it would be.

Always good to hear ;-)

> Here's what I did:
...
> class WikiLink(Transform):
> 
>     default_priority = 400
...
> Oh, and what exactly is the default_priority bit?  If I set it high,
> will Wiki links be created before internal links (hence subsuming
> internal links)?

default_priority is used for ordering the transforms.  From PEP 258
(http://docutils.sf.net/spec/pep-0258.html#transformer):

    Transform classes each have a ``default_priority`` attribute which
    is used by the Transformer to apply transforms in order (low to
    high).  The default priority can be overridden when adding
    transforms to the Transformer object.

Perhaps "priority" isn't the right term?  (Better term, anyone?)  The
transform with the lowest priority value is applied first.  There's
the connotation that "highest priority" goes first, opposite from
what's used.

See http://docutils.sf.net/spec/transforms.html for a complete list of
transforms and the order they are applied.  I would say that the
WikiLink transform should be applied after references.InternalTargets,
and therefore should have a priority value of 680.  The value of 400
used above will apply WikiLink before any of the standard hyperlink
resolution transforms, causing *all* references to be Wiki references
(i.e., none would be "resolved" prior to WikiLink seeing them).

> # Then to actually use it I had to do:
> 
> from docutils import readers
> # WikiReST is the module I gave above, and it's located here:
> sys.path.append('/usr/home/ianb/prog/WebwareWiki/Wiki')
> readers._reader_aliases['wiki'] = 'WikiReST'
> 
> # Then I use 'wiki' for the reader_name in
>   docutils.core.publish_string

That would be fine if your Reader was a module installed in
docutils/readers.  Since it isn't...

> it would be better still if I could have simply passed in the class
> to an argument to publish_string. Of course, now that I look at it,
> there's a reader argument to publish_string

Bingo!

> -- but I have to give it an instance, and that seem awkward to make
> -- how can I instantiate the reader if I have to pass the parser to
> the constructor, and the parser is created in publish_string?

The Reader will instantiate the parser automatically.  Like this::

    import WikiReST
    from docutils import core
    reader = WikiReST.Reader(parser=None,
                             parser_name='restructuredtext')
    output = docutils.core.publish_string(reader=reader,
                                          parser=reader.parser,
                                          ...)

I've just added default parameter values to the
``docutils.readers.Reader.__init__`` method to make Reader
instantiation easier::

    reader = WikiReST.Reader()

> Of course, it would be good if there was a better hook for adding a
> reader like that.

As I mentioned, that interface is only meant for installed modules.

> For instance, a better use of __import__ in
> docutils.readers.__init__ would have kept me from having to add
> something to the path

In fact, there's a to-do item that proposes removing the temptation of
using sys.path magic:

    * In ``docutils.readers.get_reader_class`` (& ``parsers`` &
      ``writers`` too), should we be importing "standalone" or
      "docutils.readers.standalone"?  (This would avoid importing
      top-level modules if the module name is not in docutils/readers.
      Potential nastiness.)

So it becomes a simple choice: get a standard (installed) component by
name, or pass a custom component object explicitly.

-- 
David Goodger  <goodger@...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

On Wed, Oct 30, 2002, David Goodger wrote:
>
> Perhaps "priority" isn't the right term?  (Better term, anyone?)  The
> transform with the lowest priority value is applied first.  There's
> the connotation that "highest priority" goes first, opposite from
> what's used.

Perhaps an entirely different system might be better.  Instead of making
ordering implict based on the priority ("order" might be a better word,
particularly since increasing numbers work well with order)), instead
have some kind of ``TransformList`` class with methods
``insert_before()`` and ``insert_after()`` that take a transform that's
the insert point (0 and -1 indicate first and last as usual), plus either
a new transform or an ordered sequence of transforms to be inserted.

People are much more likely to remember the names of transforms than
either the number or the exact order.
-- 
Aahz (aahz@...)           <*>         http://www.pythoncraft.com/

Project Vote Smart: http://www.vote-smart.org/

> On Wed, Oct 30, 2002, David Goodger wrote:
>> Perhaps "priority" isn't the right term?  (Better term, anyone?)  The
>> transform with the lowest priority value is applied first.  There's
>> the connotation that "highest priority" goes first, opposite from
>> what's used.

Aahz wrote:
> "order" might be a better word, particularly since increasing
> numbers work well with order

Perhaps, although "order" is such an overloaded term that it almost always
requires clarification.  Some other candidate terms: sequence, index,
rank/ranking, grade, position.

> Perhaps an entirely different system might be better.  Instead of making
> ordering implict based on the priority ..., instead
> have some kind of ``TransformList`` class with methods
> ``insert_before()`` and ``insert_after()`` that take a transform that's
> the insert point (0 and -1 indicate first and last as usual), plus either
> a new transform or an ordered sequence of transforms to be inserted.

I suspect that such a system would have its own problems.  To position
transform A before transform B, B would have to be registered already.  Or B
would have to know about the relationship, which may not always be the case.
The previous system, similar to the above, failed and was replaced by the
current one because it was too coarse.

An absolute numbering is very clear and simple IMO.

> People are much more likely to remember the names of transforms than
> either the number or the exact order.

Although few of us will ever have to look at the transforms, and only
infrequently.  The spec/transforms.txt file lists them all, in order, when
we do need to refer to them.

-- 
David Goodger  <goodger@...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

> > The one thing I *can't* do is help you figure out how to do this in 
> > code, because I haven't spent much time with reST in ages (okay, 
> > ever),
> 
> Garth *did* do a very cool overhaul of the unit test 
> framework, for which he has our eternal gratitude.

:)

is it correct that the numbering scheme is not taken from source.

this::

   ordered list with numbering.

   1. one
   2. two

   another list with Uppercase letters.

   A. an A
   B. a B

   how does it look like.


produces this::

   ordered list with numbering.
    1. one
    2. two

   another list with Uppercase letters.
    1. an A
    2. a B

   how does it look like.



-- 
 BINGO: completely leverage other's inexpensive benefits
 --- Engelbert Gruber -------+
 SSG Fintl,Gruber,Lassnig   /
 A6410 Telfs Untermarkt 9  /
 Tel. ++43-5262-64727 ----+

grubert@... wrote:
> is it correct that the numbering scheme is not taken from source.

No.  It works in HTML.  What output format/Writer are you talking
about?

-- 
David Goodger  <goodger@...>  Open-source projects:
  - Python Docutils: http://docutils.sourceforge.net/
    (includes reStructuredText: http://docutils.sf.net/rst.html)
  - The Go Tools Project: http://gotools.sourceforge.net/

On Fri, 8 Nov 2002, David Goodger wrote:

> grubert@... wrote:
> > is it correct that the numbering scheme is not taken from source.
>
> No.  It works in HTML.  What output format/Writer are you talking
> about?

tools/html.py, whatever it uses. i see to it closer.


--
engelbert gruber
email: engelbert.gruber@...