docstring-develop — Project development discussions.

Re: [Docstring-develop] DPS - possible bugs/features

[Ueli S. <u_s...@bl...>]

[David]
> [Ueli]
> > Now, it seems to me that the structures of documents and sections
> > are close relatives:
> > 
> > - A document may or may not have a title, a section always has one.
> 
> It is actually intended that by the time the document tree gets to the
> writer, it must have a title. The parser can't always determine the
> title by itself, such as in PySource mode. The PySource reader is
> expected to supply all the titles as appropriate.

For PySource mode, this is certainly how it should work.  However, I'm
not sure whether it is always the desired behaviour.  I believe that
standalone rtxt files will often not have a formal document title,
just a few sections.  The reader/linker can't make up a sensible guess
in this case, and IIRC one important goal is to not try to outsmart
the user.  What would be a sensible default title, anyway?  Which
leads me to believe that the document title should be left optional
(but see filename_).  BTW, that's what dps/spec/gpdi.dtd says, too::

  <!-- Optional elements may be generated by internal processing. -->
  <!ELEMENT document
      ((title, subtitle?)?, (%bibliographic.elements;)*, abstract?,
       %structure.model;)>
  <!ATTLIST document %basic.atts;>

[David]
> Basically, yes, sections are simple sub-documents. The top-level
> document does need to be special-cased in the end however. HTML pages
> need their titles!

... and other types of output can merrily do without them.  LaTeX
vs. HTML seems to be a good example here!

In LaTeX, I'd definitley use the document title as an argument to
``\title{...}`` and leave the ``\title{...}`` out if the document had
none.  In HTML, though, <title>...</title> elements aren't displayed
(AFAIK, I'm not fluent in HTML).  Not knowing what they're meant for,
I'd be perfectly comfortable with something generated in this case,
like the [filename]_ or something along the lines of "Document
generated by pydps" (which is what my pydps/html.py does).

.. [filename] The source filename isn't known to the writer, is it?
   Still, say I want ``<title>filename.rtxt</title>`` in HTML, but I
   definitely don't want ``\title{filename}`` in LaTeX.  How about
   giving the title a "generated" attribute?  Then it's left to the
   writer to use (or ignore) it, but any document could be required to
   have a title.  (Which would mean to update the DTD.)

   (BTW, my first idea was to add a "sourcefile=filename.rtxt"
   attribute to the document.  I like the "generated" much better,
   though!)

Ueli

Re: [Docstring-develop] DPS - possible bugs/features

[David G. <go...@us...>]

[Tony]
> First, I've uploaded a new pydps

Great; I'll have to take a look at it. Where should it be installed on
the source tree? (Perhaps a short README.txt file?)

Tony's problem #1 is regarding a transformation that takes place when
a document contains a single section and nothing else (with the
possible exception of comments and a bibliographic field list). The
section is 'promoted' to the document level.

Ueli's analysis is very good, spot-on. This promotion was a conscious
decision, for exactly the reasons he states. How else would you
specify a document title, especially in a standalone .rtxt file?

But I can see why it would be cumbersome when dealing with Python
source (PySource). Normally a docstring doesn't have an explicit
title; the title gets assigned from the docstring's parent object's
name. There will inevitably be odd cases where there is a leading
title, however. Perhaps this parser-specific transformation should be
made optional [#]_. Then you can treat everything as generic sections
until integration is complete.

.. [#] Along with bibliographic field list interpretation, RCS keyword
   filtering, and whatever other conveniences we dream up.

[Ueli]
> Now, it seems to me that the structures of documents and sections
> are close relatives:
> 
> - A document may or may not have a title, a section always has one.

It is actually intended that by the time the document tree gets to the
writer, it must have a title. The parser can't always determine the
title by itself, such as in PySource mode. The PySource reader is
expected to supply all the titles as appropriate.

> - A document may have a subtitle, bibliographic elements, and an
>   abstract.  A section has none of these.
> - The rest of the content follows the same model.

Correct.

> Can thus sections be treated as simpler cases of documents (instead of
> the other way round, which is how I understand your post)?  I'm not
> sure how I would exploit this, though...

Basically, yes, sections are simple sub-documents. The top-level
document does need to be special-cased in the end however. HTML pages
need their titles!

[Tony]
> 2. When processing the string module (yes, I know it isn't marked as
>    containing reST texts!), the text::
> 
>        [1:2]
> 
>    (or similar) is incorrectly identified as a link. Not a Good Thing.

``1:2`` won't be a link (as your example confirms), but ``a:b`` will
be. Why? Because according to RFC2396, 'a' could be a URI scheme (as
in 'http' or 'mailto'). The solution? Use inline literals.

[Ueli]
> Anyway, when I have something useful, I may post it here

Yes, please! Your explanation of the situation was quite eloquent.

> -- but only if everybody promises not to laugh out loud...

Never! <aghast>

-- 
David Goodger    go...@us...    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net

Re: [Docstring-develop] DPS - possible bugs/features

[Ueli S. <u_s...@bl...>]

While trying to get some LaTeX output from text files [#]_, I've
stumbled over your first bug/feature, too.  This is what I found after
playing around a bit:

- Say you've got a document that starts with a title that uses a style
  unique to the document.  (David uses titles over- and underlined
  with ``=`` in this case.)  This title will be the document title::

    ---------------------------
    ================
     Document title
    ================

    The first paragraph
    ---------------------------

  results in::

    <document name="document title">
	<title>
	    Document title
	<paragraph>
	    The first paragraph

- Now add a title in the same style.  You've got two sections, and no
  document title anymore.  With ::

    ---------------------------
    ===============
     First section
    ===============

    The first paragraph

    ================
     Second section
    ================

    Where's the document title?
    ---------------------------

  you get this::

    <document>
	<section name="first section">
	    <title>
		First section
	    <paragraph>
		The first paragraph
	<section name="second section">
	    <title>
		Second section
	    <paragraph>
		Where's the document title?

- OR add some text before this paragraph.  Same result, no document
  title any more, just section titles.  Again an example::

    ---------------------------
    Where's the document title?

    ===============
     First section
    ===============

    ...
    ---------------------------

  produces::

    <document>
	<paragraph>
	    Where's the document title?
	<section name="first section">
	    <title>
		First section
	    <paragraph>
		...

This makes sense to me, so I consider it a feature.  I'm actually not
sure how I'd be able to give a title to the document as a whole if the
parser worked as you expected (unless you special-cased the first
section level!)  (Explicitly discriminating between a document title
and regular section titles doesn't count here.)

Now, it seems to me that the structures of documents and sections are
close relatives:

- A document may or may not have a title, a section always has one.
- A document may have a subtitle, bibliographic elements, and an
  abstract.  A section has none of these.
- The rest of the content follows the same model.

Can thus sections be treated as simpler cases of documents (instead of
the other way round, which is how I understand your post)?  I'm not
sure how I would exploit this, though...

Hope I'm making sense here...

Ueli

.. [#] Not sure whether *that* is a good idea (me coding, not
       LaTeX ;-)  Anyway, when I have something useful, I may post it
       here -- but only if everybody promises not to laugh out loud...


"Tony J Ibbs (Tibs)" <to...@ls...> writes:

> Hi - two problem reports and an upload.
>
> First, I've uploaded a new pydps - it fixes some command line anomalies
> (so one can now pretty print a text file!), and the HTML output now
> makes a first pass at coping with references/footnotes/etc. (it doesn't
> yet fold in targets, because that's plainly a job for the transformer
> stage, and I don't have one of those yet!). Anyway, when run over
> reStructuredText.txt, the result is now something approaching useful.
>
> Now for the problems:
>
> 1. When trying to process reStructuredText.txt, the document produced
>    starts off like (output in "pretty mode")::
>
>       <document name="restructuredtext markup specification">
>         <title>
>           reStructuredText Markup Specification
>
>    This seems wrong to me - surely by the law of least surprise,
>    it should actually be:
>
>       <document name="restructuredtext markup specification">
>         <section>
>           <title>
>             reStructuredText Markup Specification
>
>    After all, the document starts with a title, and everywhere else
>    in the document, a title is a signal that one is starting a section.
>    (this isn't pure pedantry - it makes it a lot easier for me to
>    determine what is going on - I don't *particularly* want to special
>    case "document", and I *do* want to be able to cope well with
>    documents that *don't* start with a title...)
>

[...]

[Docstring-develop] DPS - possible bugs/features

[Tony J I. (Tibs) <to...@ls...>]

Hi - two problem reports and an upload.

First, I've uploaded a new pydps - it fixes some command line anomalies
(so one can now pretty print a text file!), and the HTML output now
makes a first pass at coping with references/footnotes/etc. (it doesn't
yet fold in targets, because that's plainly a job for the transformer
stage, and I don't have one of those yet!). Anyway, when run over
reStructuredText.txt, the result is now something approaching useful.

Now for the problems:

1. When trying to process reStructuredText.txt, the document produced
   starts off like (output in "pretty mode")::

      <document name="restructuredtext markup specification">
        <title>
          reStructuredText Markup Specification

   This seems wrong to me - surely by the law of least surprise,
   it should actually be:

      <document name="restructuredtext markup specification">
        <section>
          <title>
            reStructuredText Markup Specification

   After all, the document starts with a title, and everywhere else
   in the document, a title is a signal that one is starting a section.
   (this isn't pure pedantry - it makes it a lot easier for me to
   determine what is going on - I don't *particularly* want to special
   case "document", and I *do* want to be able to cope well with
   documents that *don't* start with a title...)

2. When processing the string module (yes, I know it isn't marked as
   containing reST texts!), the text::

       [1:2]

   (or similar) is incorrectly identified as a link. Not a Good Thing.

Hmm - taking a text file containing::

    This is a document with no title.

    What happens if we have a Python range, like [1:2] or [a:b]
    (or even [http:fred])?

and outputting it in "pretty mode" gives us::

    <document>
      <paragraph>
        This is a document with no title.
      <paragraph>
        What happens if we have a Python range, like [1:2] or [
        <link refuri="a:b">
          a:b
        ]
        (or even [
        <link refuri="http:fred">
          http:fred
        ])?

which is interesting.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

RE: [Docstring-develop] Adding pydps/pysource to CVS

[Tony J I. (Tibs) <to...@ls...>]

Garth Kidd wrote:
> If you can ssh to SourceForge, install the CygWin toolkit --
> it doesn't need administrator rights -- and use the
> SourceForge repository directly. It's pretty easy.

Ah. The problem at work isn't technical but "political". For reasons
(which I happen to agree with) of company policy, we can't just install
software on our machines without getting permission (or doing
evaluation, etc.).

   (we *do* have ssh available, though - via PuTTY on NT
   (see below))

We do *have* cygwin available on our NT machines (I believe - we
certainly have enough Unix commands available!). Of course, whilst I'm
running [my private copy of Python 2.1 - evaluation, see above] on NT,
the *directories* containing the pydps stuff are on Unix, and I tend to
use rftp on Unix to upload them to ntlworld. Humph. I may talk to Owen
(our systems admin guy - who is a friend of the guy who wrote PuTTY,
interestingly enough).

> Anyone on the list speak Debian? I'm stuck in RPM-land
> (thankfully, SuSE).

Oh, part of the problem is just that it seems to be very easy to take
initial wrong steps with CVS, and once you've done so to get thoroughly
confused. It's also that in the hour or two it takes to sort out what
one is doing, I could be doing *useful* things, like coding pydps or
bathing the kids or ironing or sleeping...

> > One possibility (but more work for someone else) might be
> > to "mirror" the files on www.tibsnjoan.co.uk onto sourceforge
> > - but I imagine that would require human intervention.
>
> Yep, but not that hard. David or I can put it up, and when
> you release a new tarball we can extract it in-place and
> commit the updates for you.

Hmm - still more work than I'd like someone else to have to do.

Leave me to think about it for the moment, and I'll get back to you (for
instance, it *may* be that Owen actually *does* have CVS around for his
own uses, and is just not making it "visible" generally, to save
confusion - internally we use RCS and some decade-or-more old stuff
written in, erm, not very good Python (not by me!)).

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

Re: [Docstring-develop] Adding pydps/pysource to CVS

[Garth K. <ga...@de...>]

> BUT unfortunately I cannot have CVS running at work, and I often do
> significant bits of development at my work machine.

If you can ssh to SourceForge, install the CygWin toolkit -- it doesn't need
administrator rights -- and use the SourceForge repository directly. It's
pretty easy.

> I guess I should explore getting CVS working at home, but my online
> connection at home is *seriously* flaky (and the first time I tried to
> install CVS, for another purpose, I got hopelessly confused - even
> Debian can't cope with a total lack of understanding!).

Anyone on the list speak Debian? I'm stuck in RPM-land (thankfully, SuSE).

> One possibility (but more work for someone else) might be to "mirror"
> the files on www.tibsnjoan.co.uk onto sourceforge - but I imagine that
> would require human intervention.

Yep, but not that hard. David or I can put it up, and when you release a new
tarball we can extract it in-place and commit the updates for you.

Regards,
Garth.

RE: [Docstring-develop] Adding pydps/pysource to CVS

[Tony J I. (Tibs) <to...@ls...>]

> Tony, how about adding pydps/pysource to CVS? Have you been able to
> get CVS access?

In theory I would love to have it in CVS (for a start, I feel somewhat
risky about continually updating with no versioning behind me).

BUT unfortunately I cannot have CVS running at work, and I often do
significant bits of development at my work machine.

I guess I should explore getting CVS working at home, but my online
connection at home is *seriously* flaky (and the first time I tried to
install CVS, for another purpose, I got hopelessly confused - even
Debian can't cope with a total lack of understanding!).

One possibility (but more work for someone else) might be to "mirror"
the files on www.tibsnjoan.co.uk onto sourceforge - but I imagine that
would require human intervention.

> We could create a "sandbox" directory for playing around, either
> beside "test" (distributed with the snapshots) or beside "web" (not
> distributed). Or put the code in a suitable place inside the DPS
> package itself.

I'm not entirely sure I understand the above, so I'll leave it alone...

Tibs (singlehandedly holding back the tides of technological progress)

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

Re: [Docstring-develop] Adding pydps/pysource to CVS

[Garth K. <ga...@de...>]

> Tony, how about adding pydps/pysource to CVS? Have you been able to
> get CVS access?

It isn't there yet.

> We could create a "sandbox" directory for playing around, either
> beside "test" (distributed with the snapshots) or beside "web" (not
> distributed). Or put the code in a suitable place inside the DPS
> package itself.

I'm -0 to peer of ``test``, +0 to it being a different module under CVS (ie
peer of ``web``); pydps *uses* dps, but it's not necessarily *part* of it.
Note that I'm not all that worried either way. :)

Regards,
Garth.

[Docstring-develop] Adding pydps/pysource to CVS

[David G. <go...@us...>]

Tony, how about adding pydps/pysource to CVS? Have you been able to
get CVS access?

We could create a "sandbox" directory for playing around, either
beside "test" (distributed with the snapshots) or beside "web" (not
distributed). Or put the code in a suitable place inside the DPS
package itself.

-- 
David Goodger    go...@us...    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net

[Docstring-develop] pydps - name/value annotation added

[Tony J I. (Tibs) <to...@ls...>]

As ever:

	http://www.tibsnjoan.co.uk/reST/pydps.tgz

This lunchtime's work is to change how names are handled, such that the
HTML can report on what a name is *set to*. This is (obviously!) limited
in usefulness, and can only be done if the name is assigned to exactly
once at the particular level it "belongs to". Amendment by methods and
functions will, of course, not be seen.

Look at html/html.py for an example of what this looks like.

It *can* be misleading, as if one does::

    a = 1
    b = a
    a = 2
    c = a

you will get "a" reported with no value (since it has more than one
assignment), but "b" and "c" both reported as being assigned "a",
despite the fact their values are different. It may be that more of the
reports need suppressing.

On the other hand, if one is doing::

    def __init__(self):
        self.postfix = 0

then being told that "self.postfix = 0" is quite useful...

(and yes, I *know* I'm reporting names in places that DPS doesn't want
me to, but information suppression isn't coded yet)

(note for anyone reading the code (yuck) that the *old* method is still
in there as well, and also that the *gathering* of this information is,
erm, rather inefficient since it's done by yet another scan of the AST)

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
.. "equal" really means "in some sense the same, but maybe not
.. the sense you were hoping for", or, more succinctly, "is
.. confused with". (Gordon McMillan, Python list, Apr 1998)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] RE: [Doc-SIG] Producing output

[Tony J I. (Tibs) <to...@ls...>]

David Goodger replied to Ueli Schläpfer:
> > - `dps2html` wants an `Errorist` class, whoch I found nowhere and
> >   assumed that David's `Reporter` would do fine.
>
> You're the victim of rapid change. I just checked this change
> in yesterday; `Reporter` is the new name. Tony hasn't had a
> chance to update his code yet.

Well, actually I changed it, but I hadn't announced it.

> What is `dps2html`? I haven't seen that one. Perhaps an old copy?

I think he's just using it as a generic term. Certainly the diffs are
with respect to an old version of pydps/html.py.

Anyway, a status report on this for Doc-SIG might not be a bad thing...

I've been working on a package to read Python modules/packages and
produce HTML therefrom. It uses Tools/compilers and dps/reST to do the
hard work. It is in a *very* alpha stage (it doesn't render lots of
things as HTML, just outputs tags in red for them, for instance, and it
definitely doesn't know everything it should about the Python code yet -
especially, it doesn't check the presence or value of ``__docformat__``,
which is naughty of it! Oh, and it has a very garish colour scheme for
the HTML(!)).

Copies are periodically uploaded as:

	http://www.tibsnjoan.co.uk/reST/pydps.tgz

but see the DPS/reST "table of links" on

	http://www.tibsnjoan.co.uk/

for a reliable pointer, as a name change is possible in the reasonably
near future (it may be becoming "pysource").

The latest version (yesterday's) does indeed now know about Reporter
rather than Errorist. Despite being aimed at Python code, it can process
a text DPS file - for instance::

    python pydps/pydps.py --text textfile.rst textfile.html

(for those following these things, significant changes are that the
command line interface now uses getopt (yuck), and I think I've now got
it representing all expression nodes, so it should be able to cope with
any function arguments that get thrown at it.)

If you're trying to learn about what DPS/reST *does*, then the command::

    python pydps/pydps.py --text textfile.rtxt

produces rather nice output (although "quicktest.py" can produce the
same), and if you want to add in the structure of the (current) Python
information (subject to vast change at zero notice, though - I'm afraid
on a project like this, the ability to refactor (gosh, I used to just
call that "change"!) Python code quickly and safely is very useful),
try::

    python pydps/pydps.py --pretty pythonfile.py

There's zero documentation (well, there's a bit if you run it on
itself!) other than the ``--help`` command. It requires one to have
installed the latest versions of dps and restructuredtext, and the
Tools/compiler for your Python (trying to use the wrong one may lead to
odd effects) - all neat things to have around anyway (!). It is not
expected to work on Python before 2.0.

> > In pydps/html.py, there are several references to `element.tagName`
> > which should probably read `element.tagname`.
>
> You must be looking at an old version. Tony's latest has been
> updated to this particular change.

Yes, days ago (maybe even a week or more).

> If you're interested (and I hope you are!),

I'll second that - the more people playing (and *using* counts as
playing for this game!) the better.

Note that I've been announcing significant changes in pydps over on the
docstring-develop list:

	http://lists.sourceforge.net/lists/listinfo/docstring-develop

rather than on the main Doc-SIG (when I remember), partly because the
changes are too frequent (and unfortunately I'm not integrated with the
CVS tree).

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

RE: [Docstring-develop] Re: pydps - function signatures and Tools/compiler questions

[Tony J I. (Tibs) <to...@ls...>]

Jeremy Hylton responded:
> >>>>> "TJI" == Tibs  <Tony> writes:

s'OK - I answer indiscriminately to my usename and my bornname.

> The LaTeX source is the right place to make changes.

OK. I'll try to have a go "at some stage".

> If you've got updates, it's probably best to use the patch
> manager to submit changes.

Oh dear - another new thing to learn. Ah well.

>   TJI> 1. Is there a possibility that the compiler module might get
>   TJI> shifted from Tools into the standard library? It seems rather
>   TJI> too useful to be stuck "in the hinterlands".
>
> I think it's primarily a question of user demand.  If people are using
> it enough, it might move.

My thinking was more that it is going to be fundamental to pydps (or
whatever it gets called), and it's a pain if we have to get people to
install something else (which they might not have around - Tools doesn't
come with the Windows installer, I think).

>   TJI> 2. Should the various representation functions I'm working on
>   TJI> eventually be merged in as methods on the compiler nodes?
>   ...deletia...

> I think you can count on the organization of the ast classes staying
> basically the same.  The names of the classes and the attributes of
> their instances are a crucial part of the API.  Is that sufficient?

Hmm - maybe. I'll try to work up a "proper" description of anything I
think might benefit the innards of the compiler module as such, if I do
decide it isn't.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Well we're safe now....thank God we're in a bowling alley.
- Big Bob (J.T. Walsh) in "Pleasantville"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] Re: pydps - function signatures and Tools/compiler questions

[Jeremy H. <je...@zo...>]

>>>>> "TJI" == Tibs  <Tony> writes:

  TJI> Jeremy - this means that I'm getting a *much* better
  TJI> understanding of some of the nodes in the compiler tree. I
  TJI> realise that the table in

  TJI> 	http://www.python.org/~jeremy/compiler/module-compiler.ast.html

  TJI> is generated from LaTeX source, presumably that at

  TJI> http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/To
  TJI> ols/compiler/doc/asttable.tex

That's right. 

  TJI> (well, when I browse it). I'm willing to update that, in
  TJI> whatever form is useful, with more information about what each
  TJI> "entry" is. What's the best way for me to do that, so far as
  TJI> you're concerned? (taking a copy of the LaTeX and working on it
  TJI> would be OK by me).

The LaTeX source is the right place to make changes.  If you've got
updates, it's probably best to use the patch manager to submit
changes.

  TJI> Two other questions:

  TJI> 1. Is there a possibility that the compiler module might get
  TJI> shifted from Tools into the standard library? It seems rather
  TJI> too useful to be stuck "in the hinterlands".

I think it's primarily a question of user demand.  If people are using
it enough, it might move.

  TJI> 2. Should the various representation functions I'm working on
  TJI> eventually be merged in as methods on the compiler nodes? I'm a
  TJI> bit chary of having code outwith the compiler module that has
  TJI> to know details about a good number of the nodes inside (e.g.,
  TJI> the problems if a new class gets added). On the other hand,
  TJI> cruft that (maybe) no-one else would want would be a bad
  TJI> thing...

I think you can count on the organization of the ast classes staying
basically the same.  The names of the classes and the attributes of
their instances are a crucial part of the API.  Is that sufficient?

Jeremy

[Docstring-develop] pydps - function signatures and Tools/compiler questions

[Tony J I. (Tibs) <to...@ls...>]

It's still

      	http://www.tibsnjoan.co.uk/reST/pydps.tgz

I've started adding support for function signatures. This has meant
working on producing representations for the RHS of assignments, which
is partially completed (it does, for instance, list comprehensions, but
not basic arithmetic). See my other message on the Doc-SIG as such today
(Re: [Doc-SIG] DPS DTDs) for an example of what it handles.

Jeremy - this means that I'm getting a *much* better understanding of
some of the nodes in the compiler tree. I realise that the table in

	http://www.python.org/~jeremy/compiler/module-compiler.ast.html

is generated from LaTeX source, presumably that at


http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/To
ols/compiler/doc/asttable.tex

(well, when I browse it). I'm willing to update that, in whatever form
is useful, with more information about what each "entry" is. What's the
best way for me to do that, so far as you're concerned? (taking a copy
of the LaTeX and working on it would be OK by me).

Two other questions:

1. Is there a possibility that the compiler module might get shifted
from Tools into the standard library? It seems rather too useful to be
stuck "in the hinterlands".

2. Should the various representation functions I'm working on eventually
be merged in as methods on the compiler nodes? I'm a bit chary of having
code outwith the compiler module that has to know details about a good
number of the nodes inside (e.g., the problems if a new class gets
added). On the other hand, cruft that (maybe) no-one else would want
would be a bad thing...

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
You said "run as root" and "securely" in the same sentence relating to
CGI. You're funny! -- Ignacio Vazquez-Abrams, on the Python list
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] RE: [Docstring-checkins] CVS: dps/dps nodes.py,1.7,1.8

[Tony J I. (Tibs) <to...@ls...>]

>   - Changed 'tagName' to 'tagname'.

Drat - I'll have to edit some code (actually, only a minor
inconvenience, but it's interesting that this is the first change to DPS
for *ages* that has required me to change anything).

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] pydps - package support

[Tony J I. (Tibs) <to...@ls...>]

And again,

      	http://www.tibsnjoan.co.uk/reST/pydps.tgz

Well, I should have been working, but instead I've added rudimentary
package support to pydps. The above contains a new directory within the
pydps directory, html, which contains two example outputs, produced by
cd'ing to the pydps directory and doing::

    python pydps.py html html.py > html/html.html
    python pydps.py html ..\pydps > html/pydps-package.html

(yes, I'm afraid it was on Windows NT).

Since "verbose" is enable by default, if you try those you'll get *lost*
of stuff printed to stderr - never mind.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Which is safer, driving or cycling?
Cycling - it's harder to kill people with a bike...
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] pydps - nothing very exciting

[Tony J I. (Tibs) <to...@ls...>]

Ho hum, bla bla,

     	http://www.tibsnjoan.co.uk/reST/pydps.tgz

I finally fixed the stupid bugs I'd left in sorting out the left
alignment of docstrings (before they get handed to DPS). That's what I
get for programming before bedtime.

The "dumb" HTML formatter is now working, and is split into a Writer
class (which knows about some of the DPS nodes) and a PythonWriter
subclass (which knows about some of the Python "outer level"
infrastructure).

It handles enough of reST to be usefully run on its own modules
(packages aren't supported yet, I'm afraid) and get a reasonably
coherent result.

I seriously like the paradigm of "instantiate a Writer class once and
then call it as needed" - this allows good encapsulation of the baggage
one wants to define only once for a multiplicity of documents, and also
allows one to reduce the number of arguments to the "write" call without
needing scads of optional arguments. I still remember how neat I
realised the __call__ method was when I finally internalised its use.

Speaking of which (hmm - this is probably a Doc-SIG question, but
still), which "special" (i.e., __xxx__) methods do we want to *normally*
report on with DPS? Obviously __init__ and __call__ are extra special,
since their arguments matter. (of course, *removing* things from the
output is unlikely to happen until a long way down the line - so it's
not a decision that's needed Real Soon Now).

Next thing is to transfer more information across from visit.py (where I
remember the information derived from the AST) to nodes.py (where I
place it into a DPS nodes tree) and thus make it available to html.py
(for presentation).

[of course, the school year has now started again, so mornings are now
more rushed getting Michael to school, and evenings have less time, so
Glacial Software Development is back in action, I'm afraid...]

Tibs

(ps: thanks for the nightly tarballs - that makes my life a LOT neater -
a couple of times I've noticed a bug, and magically had it go away the
next day - great stuff!)

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
You said "run as root" and "securely" in the same sentence relating to
CGI. You're funny! -- Ignacio Vazquez-Abrams, on the Python list
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] pydps - the starts of HTML output

[Tony J I. (Tibs) <to...@ls...>]

[Neal, Jeremy - do you still want me to CC you, or shall I drop back to
just the docstring-develops list? I'll take no answer as meaning make no
change...]

Had fun this weekend getting the quick-and-dirty HTML output started. In
the end, I've stopped faffing around with DOM trees (all say "yeah!")
and am using the dps/nodes.py tree (DPS nodes tree?) directly - it's
pleasantly simple to use.

As always, latest version is at:

    	http://www.tibsnjoan.co.uk/reST/pydps.tgz

As you might expect (!) the colours are garish - this is my oh-so-subtle
way of indicating that this is a PROTOTYPE.

Still to do:

* add some more functions to the html.py, so it can cope
  with more stuff

* expand nodes.py to transfer more of my initial data
  into the DPS nodes tree.

* expand visit.py to work properly (!) - various things
  need doing - see the file.

* deal with cross-references.

* add more documentation in various places.

* extract the command line stuff to its own file, make
  the directory a proper package, etc., etc.

* integrate as a basic DPS Formatter, plus extra subclass
  to handle the Python stuff.

As you can see, there's still plenty of fun in store (I estimate at
least another two major refactorings, for a start).

For what its worth, I'm finding the DPS (and also the compiler package)
good fun to play with - it's nice to leave someone else to handle the
parsing!

Tibs
--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"Bounce with the bunny. Strut with the duck.
 Spin with the chickens now - CLUCK CLUCK CLUCK!"
BARNYARD DANCE! by Sandra Boynton
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] dps_visit.py - first pass at XML output

[Tony J I. (Tibs) <to...@ls...>]

As always, they're at:

   	http://www.tibsnjoan.co.uk/reST/dps_visit.py
   	http://www.tibsnjoan.co.uk/reST/test.py

The addition is an "xml" command line switch, which causes a (rather
nasty) emission of XML to the standard output. No attempt to process the
docstrings yet. No attempt to think about a nice schema for the XML,
either.

Note that this module now depends on the DPS module, since it imports
(its of) it. I fully intend to produce a separate module at some stage
which is useful *just* as an example of mining the relevant information
out of the AST - that should simply be a matter of stripping stuff out.

NB: Jeremy - do we believe the compiler module to work under 2.2a1? I
had a problem last night that I avoided by using an earlier version of
it - sorry, I don't have the exception stack to hand, and unfortunately
I don't have internet connectivity at home at the moment, so can't
download 2.2a2 (which, for all I know, may fix it). I'll try to research
more tonight, in case this is something not known about.

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] Latest dps_visit.py

[Tony J I. (Tibs) <to...@ls...>]

New versions of:

  	http://www.tibsnjoan.co.uk/reST/dps_visit.py
  	http://www.tibsnjoan.co.uk/reST/test.py

- I've added detection of globals that are declared and used below the
top level.

For instance (hope this doesn't wrap):

U:\dps>python dps_visit.py show test.py
Parsing file test.py
Locating attribute docstrings

Module test (in file test.py)
  """A simple test file for input to ast-mining.py.
  """
  Import string
  From string import lstrip, rstrip
  Attributes
    one
      """Documentation after `one` (intervening blank line)
      """
    two
    six
      """Documentation after ``six=[four,five]``
      """
    gmodule
      """Global at the module level
      """
    three
    five
    four
  Class Fred
    """Class at the module level.
    """
    Method __init__(self)
      """Initialisation for `Fred`
      """
      Attributes
        b
          """`self.b` within `Fred.__init__()`
          """
      Class Insider
        """Class defined inside `Fred.__init__()`
        """
        Method fred(self)
          Function fredinner(())
  Class Jim(Fred)
  Function func(a, b=1, c='jim', d=None, e=List(()))
    """Function at the module level
    """
    Class InFunc
    Function infunc(())
      """Function defined inside function `func()`
      """
      Class Silly
      Function silly_fn(())
Globals defined *and used* below the top level:
  Global gagain     used in Method   '__init__'   in Class    'Fred'
                     and in Function 'fredinner'  in Method   'fred'
  Global gclass     used in Class    'Fred'       in Module   'test'
  Global gclass2    used in Class    'Insider'    in Method   '__init__'
  Global gfunc      used in Function 'func'       in Module   'test'
  Global gfunc2     used in Method   'fred'       in Class    'Insider'
  Global ginner     used in Function 'infunc'     in Function 'func'
  Global ginner2    used in Function 'fredinner'  in Method   'fred'
  Global gmeth      used in Method   '__init__'   in Class    'Fred'

U:\dps>

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] dps_visit.py (was ast-mining.py)

[Tony J I. (Tibs) <to...@ls...>]

This time, the files are:

 	http://www.tibsnjoan.co.uk/reST/dps_visit.py
 	http://www.tibsnjoan.co.uk/reST/test.py

ast-mining.py was becoming too unspecific, and as Jeremy pointed out,
not a valid name for importing.

I've changed various odds and ends - the scope info passed down now
means that functions inside functions inside... know about their scope
and superscope, which gives some hook for globals in them.

The next things to do are:

1. Internal documentation
2. Collect information about globals
3. Prune the final tree for DPS use
4. Plug in some DPS/reST code and see what happens

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

[Docstring-develop] ast-mining version 0.3

[Tony J I. (Tibs) <to...@ls...>]

I've updated:

	http://www.tibsnjoan.co.uk/reST/ast-mining.py
	http://www.tibsnjoan.co.uk/reST/test.py

A first approximation to finding (DPS required) attribute docstrings is
in place (it's either very icky or very elegant - possibly both at
once).

Next is to make it aware of "hidden" globals - so the poor user can be
warned that there is a usage like::

    class A:
        def B():
            def C():
                def D():
                    global Z
                    # 100 lines of code, not mentioning Z
                    # and Z is not at the top level either
                    Z = 100

(which I think would be a Useful Thing to do - and also it's an
interesting thing to work out how to do).

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
"How fleeting are all human passions compared with the massive
continuity of ducks." - Dorothy L. Sayers, "Gaudy Night"
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)

Re: [Docstring-develop] Attribute docstrings

[David G. <dgo...@bi...>]

Tony J Ibbs (Tibs) <to...@ls...> wrote on 2001-08-16 09:42:
> I'm sending this to the development group and not the doc-sig
> because I'm not convinced it makes enough sense!

You're not convinced it makes sense to send this to the larger Doc-SIG
group, or you're not convinced that the *message itself* makes sense?
If the former, I think Doc-SIG *would* be appropriate (possibly get a
wider range of input). If the latter, I can't help you. ;-)

Seriously, I think docstring-develop should be used for
implementation issues (bugs, etc.), and Doc-SIG should be used for
general issues (policy, syntax). I've replied to Doc-SIG.

/David

Re: [Docstring-develop] Further adventures of ast-mining.py

[David G. <go...@us...>]

Tony J Ibbs (Tibs) <to...@ls...> wrote on 2001-08-16 09:18:
> [recipients pruned - I've removed Python-Dev - will they care?]

I think it's a good call.

> I've uploaded two files:
> 
>     http://www.tibsnjoan.co.uk/reST/ast-mining.py
>     http://www.tibsnjoan.co.uk/reST/test.py

Looks promising!

-- 
David Goodger    dgo...@us...    Open-source projects:
 - Python Docstring Processing System: http://docstring.sourceforge.net
 - reStructuredText: http://structuredtext.sourceforge.net
 - The Go Tools Project: http://gotools.sourceforge.net

[Docstring-develop] Attribute docstrings

[Tony J I. (Tibs) <to...@ls...>]

I'm sending this to the development group and not the doc-sig because
I'm not convinced it makes enough sense!

DPS (pep 258) is proposing to support attribute docstrings.

Yet PEP 224 was rejected by the BDFL because he dislikes the look of::

    a = 1
    """Documentation."""
    b = 1

a. because he doesn't like it
b. because he can't tell if the docstring belongs
   to `a` or `b`

This rejection did *not* worry about issues of what value was used to
store the docstring (obviously not an issue for us).

Is DPS on dodgy ground in supporting a PEP that has been fundamentally
rejected?

As a potential *slight* cure for the BDFL objections (I'm doubtful), I
can only propose that we mark up the docstring itself::

    a = 1
    """.. doc::a Some documentation"""

would be documentation for `a`. That's a bit verbose, so maybe it should
be::

    """:doc:a Some documentation"""

Or even::

    """:a: Some documentation"""

(is that last stretching too far?)

[I'm not sure I like any of the above, in fact - do we have a BDFL
comment on this part of the DPS spec?]

Or maybe we should enforce no blank line before the docstring, and at
least one blank line after it (I've yet to see if we can tell that from
the AST, but I suspect we can).

Tibs

--
Tony J Ibbs (Tibs)      http://www.tibsnjoan.co.uk/
Give a pedant an inch and they'll take 25.4mm
(once they've established you're talking a post-1959 inch, of course)
My views! Mine! Mine! (Unless Laser-Scan ask nicely to borrow them.)