Thread: [Epydoc-devel] Planning for 3.0
Brought to you by:
edloper
From: Daniele V. <dan...@gm...> - 2007-02-05 03:52:33
|
ulimit -c unlimited... Hello Edward, what about setting a laundry list of things we want to be fixed to roll out a beta version of Epydoc 3.0? I think the greatest part of the work is done: Epydoc is already something that does things no other Python API generator does - by far (mostly thanks to all your efforts into the parsing/introspecting system). Here is a laundry list of things i'd like to discuss AND to code if this would concretely bring us near to the release. Feel free to add/delete/update items (all the __methods__ have been implemented :) - exclusion list (SF #1209634) would help big project to adopt Epydoc (just committed in the trunk a first version) - pretty printing (see also SF #1628044). How to represent an object value? - repr() is fine only for int and for classes that specifically return a parsable representation, such as datetime. Even floats (such as 0.1) often have an ugly representation. - fall back on the parsed version if available and the repr() result matches an ugly pattern looks like "<something>"? - fall back on str()ingification? - prettyprinting of lists, dicts etc? More clever truncation of long representations? - if a variable value fits on a single row, omit the detail box? - ??? - parsing some "#:" comments into functions body as they were part of the function docstring. It would keep some additional information close to the code implementing it, making the task to keep the API consistent with the code easier. Es. def getDigitsNumber(num): """Return the number of digits of a number.""" ...there may be other code #: :warning: the current implementation only allows #: positive numbers return floor(log(num) / log(10)) + 1 Some tags (note, bug, warning, todo) are really best placed next the code to be documented; precondition and postcondition would be nice close to an assert etc. - using some module __variable__ as a tag (only the ones widely used: __version__ above all, but we may be friendly with __author__ etc; people has already been warned not to abuse of such variables, though i couldn't find a reference) - docutils :python: directive to allow examples to appear as colored Python code - :api: role to allow docutils documents to refer to Epydoc generated API. See for example http://www.initd.org/tracker/psycopg/browser/psycopg2/trunk/scripts/ext2html.py - more compact html code, for which the exp-compact-html branch is open. In some personal audits i had, somebody complained about the signal/noise ratio in HTML output. Together with setting a goal in terms of features, i'd love to QA Epydoc against the most prominent OS projects, such TurboGears (i just read they are having documentation issues - cfr. http://groups.google.it/group/turbogears/browse_thread/thread/ae0e52ff3f9fde60?hl=en for example). I was very sad when i read (just a couple of weeks ago, but the issues were older) how Twisted guys were disappointed by Epydoc no more fulfilling their needs, up to decide to write their own API generator (http://codespeak.net/~mwh/pydoctor/). I'd like to check Epydoc against a list of packages, including some not specifically thought to be inspected by Epydoc. The goal is: - Epydoc must not crash (modulo the most magic corners, which now can be removed from the list of modules to be introspected) - The generation result shall be presented to the packages developers, both to gather feedback about Epydoc output and to advertise Epydoc to the community. Eventually making people happy to write documentation... A list of beasts to test against may include: - WxPython - Django - Turbogears - Twisted - SqlAlchemy - docutils Daniele ...core dumped |
From: Edward L. <ed...@gr...> - 2007-02-06 01:50:59
|
Daniele Varrazzo wrote: > what about setting a laundry list of things we want to be fixed to roll > out a beta version of Epydoc 3.0? Sounds good. One item is to go through the bug list & respond to all the bug reports -- it looks like you've already been making some good progress on that front. Unfortunately, I probably won't have much time to work on epydoc until the end of February, since I'm trying to finish up my phd proposal. But I'm happy letting you take the lead in putting together the beta and/or 3.0 release, if you're up for it. > Here is a laundry list of things i'd like to discuss [...] All of these sound like good features. I don't see any of them as being necessary for the 3.0 release, but I'd be happy to see any of them added. > - exclusion list (SF #1209634) would help big project to adopt Epydoc > (just committed in the trunk a first version) This definitely seems like a good thing. > - pretty printing (see also SF #1628044). How to represent an object > value? > - repr() is fine only for int and for classes that specifically > return a parsable representation, such as datetime. Even floats > (such as 0.1) often have an ugly representation. Yeah -- people are very inconsistent about how they use repr/str. > - fall back on the parsed version if available and the repr() result > matches an ugly pattern looks like "<something>"? Sometimes the parsed version isn't very trustworthy.. E.g., if they create a variable with a list, and then append to it, etc. But you may be right that this may be the best way to go when you get "<something>". It's almost certainly the best way if you get "<xyz instance at 0xaddr>". > - fall back on str()ingification? Yeah, it might be good to use str() for some values. > - prettyprinting of lists, dicts etc? More clever truncation of long > representations? Seems like good ideas. > - if a variable value fits on a single row, omit the detail box? Sure. > - ??? One issue/question is that right now a good amount of this is handled in specific docwriter modules.. E.g., the docwriter.html code to colorize regexps. Should this be abstracted away somehow and handled more centrally? E.g., the centralized code could generate an abstract structure saying how to color the regexp, and the individual writers would just translate that to the appropriate markup? > - parsing some "#:" comments into functions body as they were part of > the function docstring. It would keep some additional information > close to the code implementing it, making the task to keep the API > consistent with the code easier. Es. This is a little bit problematic, because it would be hard to tell the difference between "#:" comments that are supposed to describe the enclosing object vs "#:" comments that are supposed to describe a variable assignment. E.g., I think people would do things like: def __init__(self, foo): """This is a constructor of some sort.""" ...there may be other code #: :warning: the current implementation only allows #: positive numbers val = floor(log(num) / log(10)) + 1 ... or: class A: "..." ... other code #: :warning: ... x = 123 Where it's unclear whether whether that warning applies to the constructor function or to the 'val' variable. > Some tags (note, bug, warning, todo) are really best placed next the > code to be documented; precondition and postcondition would be nice > close to an assert etc. True, but I'm not sure how to reconcile this with the ability to use "#:" comments to describe variables. > - using some module __variable__ as a tag (only the ones widely used: > __version__ above all, but we may be friendly with __author__ etc; > people has already been warned not to abuse of such variables, though > i couldn't find a reference) This would be fine. It would be nice if this could be customized, sort of how @field names can currently be added with @deffield. > - docutils :python: directive to allow examples to appear as colored > Python code That would be fine. Is there a reason not to use doctest block? They already get colored. > - :api: role to allow docutils documents to refer to Epydoc generated > API. See for example > http://www.initd.org/tracker/psycopg/browser/psycopg2/trunk/scripts/ext2html.py This would need some sort of configuration to tell epydoc where that :api: role should point to. Would this be done in the config file/command line? > - more compact html code, for which the exp-compact-html branch is > open. In some personal audits i had, somebody complained about the > signal/noise ratio in HTML output. Can you point me at an example page that demonstrates the changes made by the branch? > Together with setting a goal in terms of features, i'd love to QA Epydoc > against the most prominent OS projects, such TurboGears (i just read > they are having documentation issues - cfr. > http://groups.google.it/group/turbogears/browse_thread/thread/ae0e52ff3f9fde60?hl=en > for example). This would be excellent, and *very* much appreciated. In fact, I think it might be a good idea to set up some scripts on a server somewhere to automatically run epydoc on various projects, to at least make sure that it doesn't crash. I have a server I could probably put this on (I don't think there'd be enough space on the sourceforge server -- there's a 100mb quota). This would be something like the python buildbot, but simpler of course. :) > I was very sad when i read (just a couple of weeks ago, but the issues > were older) how Twisted guys were disappointed by Epydoc no more > fulfilling their needs, up to decide to write their own API > generator (http://codespeak.net/~mwh/pydoctor/). That's too bad. :-/ Hopefully once 3.0 comes back, we can win back some of the projects that moved on to other tools. > I'd like to check Epydoc against a list of packages, including some not > specifically thought to be inspected by Epydoc. The goal is: > - Epydoc must not crash (modulo the most magic corners, which now can > be removed from the list of modules to be introspected) > - The generation result shall be presented to the packages developers, > both to gather feedback about Epydoc output and to advertise Epydoc > to the community. Eventually making people happy to write > documentation... > > A list of beasts to test against may include: > > - WxPython > - Django > - Turbogears > - Twisted > - SqlAlchemy > - docutils Sounds great. Of course the python stdlib should be included too. :) Once we do the beta release, we should make a point of encouraging people to try it out on their project & get back to us about what they think of the results. -Edward |
From: Daniele V. <dan...@gm...> - 2007-02-07 01:52:08
|
Hello, >> - docutils :python: directive to allow examples to appear as colored >> Python code > > That would be fine. Is there a reason not to use doctest block? They > already get colored. I am trying to add a directive to parse a docstring such as the following, generating a box with colored syntax:: class Foo: """A class like nothing else. This is a test: .. code:: python def foo(): '''Hello world''' for i in range(10): print i """ i'd like to generate a doctest block, so that the visitors would already work. No luck up to now: i tried to follow the example in [1], but it fails. The directive i added is:: class CodeDirective(Directive): required_arguments = 0 optional_arguments = 1 has_content = True def run(self): self.assert_has_content() text = '\n'.join(self.content) node = docutils.nodes.doctest_block(rawsource=text) self.state.nested_parse(self.content, self.content_offset, node) return [ node ] directives.register_directive('code', CodeDirective) but trying to parse the above snippet i receive errors "Unexpected indentation". It seems that "nested_parse" parses the snippet:: def Foo(): '''Hello world''' for i in range(10): print i as it were ordinary text, not a block, so indentations don't make sense for it. It should really be parsed as a literal block. I checked the docutils.parsers.rst.states module, but didn't find any promising method to "parse" a literal block (for which, i know, there's not much to parse) Any hint? Thanks -- Daniele [1] http://docutils.sourceforge.net/docs/howto/rst-directives.html#admonitions |
From: Edward L. <ed...@gr...> - 2007-02-07 02:07:59
|
Daniele Varrazzo wrote: > I am trying to add a directive to parse a docstring such as the following, > generating a box with colored syntax:: [...] I believe this should do what you want: def run(self): self.assert_has_content() text = '\n'.join(self.content) node = docutils.nodes.doctest_block(text, text) return [ node ] (The class doctest_block inherits its constructor from FixedTextElement. The first arg of the constructor gives the rawsource, and the second gives the text content.) You should only be using nested_parse if the directive contains something that is formatted using rst syntax. -Edward |
From: Daniele V. <pi...@de...> - 2007-02-10 22:33:57
|
>> - docutils :python: directive to allow examples to appear as colored >> Python code > > That would be fine. Is there a reason not to use doctest block? They > already get colored. The "python" directive has been added into the trunk. It uses the recently refactored HTMLDoctestColorizer to perform its job. An example: .. python:: # This is some Python code def foo(): pass class Foo: def __init__(self): pass -- Daniele Varrazzo - Develer S.r.l. http://www.develer.com |
From: Edward L. <ed...@gr...> - 2007-02-13 20:54:42
|
Daniele brought up the issue of improving epydoc's pretty-printing of object values. As of svn revision 1477, I've made most of the improvements that were suggested. In particular: * I refactored the pyval_repr() code, so that now syntax-highlighting is done in a central place (epydoc.markup.pyval_repr), and doesn't need to be repeated for each output format. In particular, APIDoc.pyval_repr() now returns a subclass of ParsedEpydocDocstring. * The new code does pretty-printing of lists, dicts, strings, tuples, sets, frozensets, and regexp objects. This pretty-printing should be faster than using the pprint module, in that it short-circuits as soon as the the output contains max_lines lines. It also does automatic line-wrapping. * str() is used instead of repr() for floats, ints, longs, and complex numbers. Other types could be added to this list, if desired. * When the pretty-printed representation is constructed, a score is calculated that evaluates "how useful" the representation is. If it's too low, then the parse-based representation is used instead (assuming it's available). In particular, if the string returned by an object's __repr__ has the form <.* at 0xaddr>, then epydoc will tend to use the parse-based repr instead. -Edward |
From: Daniele V. <dan...@gm...> - 2007-02-06 13:15:29
|
Edward Loper ha scritto: > Daniele Varrazzo wrote: >> what about setting a laundry list of things we want to be fixed to roll >> out a beta version of Epydoc 3.0? > > Sounds good. One item is to go through the bug list & respond to all > the bug reports -- it looks like you've already been making some good > progress on that front. > > Unfortunately, I probably won't have much time to work on epydoc until > the end of February, since I'm trying to finish up my phd proposal. But > I'm happy letting you take the lead in putting together the beta and/or > 3.0 release, if you're up for it. > >> Here is a laundry list of things i'd like to discuss [...] > > All of these sound like good features. I don't see any of them as being > necessary for the 3.0 release, but I'd be happy to see any of them added. I also thought that saying "i want the features x and y" and "i want 3.0 rolled out" means going nowhere :) so, many "bright ideas" are probably perfect... for the release 3.1. Getting back to 3.0: >> - pretty printing (see also SF #1628044). How to represent an object >> value? I don't expect this being a showstopper for 3.0, and best result can be achieved incrementally. >> - parsing some "#:" comments into functions body as they were part of >> the function docstring. [...] > > This is a little bit problematic, because it would be hard to tell [...] 3.1 :) >> Some tags (note, bug, warning, todo) are really best placed next the >> code to be documented; precondition and postcondition would be nice >> close to an assert etc. > > True, but I'm not sure how to reconcile this with the ability to use > "#:" comments to describe variables. 3.1! >> - using some module __variable__ as a tag (only the ones widely used: >> __version__ above all, but we may be friendly with __author__ etc; >> people has already been warned not to abuse of such variables, though >> i couldn't find a reference) > > This would be fine. It would be nice if this could be customized, sort > of how @field names can currently be added with @deffield. I'll start with a map variable name -> tag, then think about its customization. >> - docutils :python: directive to allow examples to appear as colored >> Python code > > That would be fine. Is there a reason not to use doctest block? They > already get colored. I thought doctest blocks require >>> and ... things in front of the lines. I already wanted to leverage markup.doctest: i'll check if there is some refactoring to perform in order to add the directive. >> - :api: role to allow docutils documents to refer to Epydoc generated >> API. See for example >> http://www.initd.org/tracker/psycopg/browser/psycopg2/trunk/scripts/ext2html.py >> > > This would need some sort of configuration to tell epydoc where that > :api: role should point to. Would this be done in the config > file/command line? Probably a command line switch for a wrapper around rst2html >> - more compact html code, for which the exp-compact-html branch is >> open. In some personal audits i had, somebody complained about the >> signal/noise ratio in HTML output. > > Can you point me at an example page that demonstrates the changes made > by the branch? Mmm.. i sent you a couple of mail that got unnoticed back on september 2006. Anyway... Quoting such message: > I fiddled with the HTML output and the default Epydoc css trying to > optimize the vertical space. A result can be seen at > http://www.develer.com/~piro/api-compact/epydoc.apidoc.APIDoc-class.html > > I'd like your opinion about the supposed problem and the proposed layout. > > Notice that i had to update not only the css, but also some html code. > Actually it looks to me kinda "old", with many alignment tricks that > nowadays can be better accomplished using CSS. For example, there are > many definitions list with empty definitions, i guess only used to > increase the left margin. I cleaned up them, minimizing the number of > warnings issued by Tidy, and using CSS to reposition them. > > For example i replaced > > <dl><dt></dt><dd> > <p><strong>Warning:</strong> > This method uses undocumented data structures inside of > <code>profile_stats</code>. > </p> > </dd></dl> > > with > > <div class="fields"> > <p><strong>Warning:</strong> > This method uses undocumented data structures inside of > <code>profile_stats</code>. > </p> > </div> > > (attached, the full patch up to now) > > If you prefer to keep the pages appearance as is (wide), then we could > provide an optional "compact" CSS. Anyway, to allow wider CSS > positioning freedom, some html cleanup may be required (using more > divs and spans instead of p's and tables, for example) The API i generated then are available at: http://www.develer.com/~piro/api/epydoc.apidoc.APIDoc-class.html http://www.develer.com/~piro/api-compact/epydoc.apidoc.APIDoc-class.html >> Together with setting a goal in terms of features, i'd love to QA Epydoc >> against the most prominent OS projects, such TurboGears (i just read >> they are having documentation issues - cfr. >> http://groups.google.it/group/turbogears/browse_thread/thread/ae0e52ff3f9fde60?hl=en >> for example). > > This would be excellent, and *very* much appreciated. In fact, I think > it might be a good idea to set up some scripts on a server somewhere to > automatically run epydoc on various projects, to at least make sure that > it doesn't crash. I have a server I could probably put this on (I don't > think there'd be enough space on the sourceforge server -- there's a > 100mb quota). This would be something like the python buildbot, but > simpler of course. :) The society i'm currently working for is an enthusiast sustainer of open source software and will be glad to offer web space to host such results. >> A list of beasts to test against may include: >> >> - WxPython >> - Django >> - Turbogears >> - Twisted >> - SqlAlchemy >> - docutils > > Sounds great. Of course the python stdlib should be included too. :) Of course, the Python lib and Epydoc itself are in the list :) > Once we do the beta release, we should make a point of encouraging > people to try it out on their project & get back to us about what they > think of the results. My plan is now: - attack the remaining bugs. I'll put the older ones in pending state asking the reporters to check out the trunk and verify if they magically disappeared: if not, the bug will be fixed. - Open a 3.1 group in the feature request tracker and move there what doesn't stop us to release a good 3.0. In the tracker i will add items from: - the TODO list in epydoc.__init__.py - my laundry list items (together with the comments you did about them) and categorize them and the currently open requests splitting between 3.0 and 3.1. - Close the 3.0 requests and declare "beta" - Spend some time parsing other open source projects and publishing the results, contributing back a cfg file to let them generate the docs by themselves - If no big issue raises from such phase, go 3.0 - Enjoy - Attacking the request list for 3.1... Regards, Daniele |
From: Edward L. <ed...@gr...> - 2007-02-09 18:18:28
|
On Feb 6, 2007, at 8:14 AM, Daniele Varrazzo wrote: >> I fiddled with the HTML output and the default Epydoc css trying to >> optimize the vertical space. A result can be seen at >> http://www.develer.com/~piro/api-compact/epydoc.apidoc.APIDoc- >> class.html >> [...] Sorry, I guess I missed this message the first time around. This looks like an improvement on the current code, and I see no problems with merging the compat branch into the trunk. I think that the reason I used <dl> rather than <div>+css is that when I started writing epydoc, css was not yet supported by several common browsers; so I wanted to generate HTML code that would display well on older browsers. Now that several years have passed, I don't think that's an issue any more -- all modern browsers should support at least basic css. So feel free to update the html writer to generate output that can be customized more easily w/ css. As for the question of how compact the default output should be, I agree that the current output has a pretty large amount of whitespace in the output, but the api-compact page you pointed at seems a little *too* tight to me. Perhaps we could change the default to be somewhat intermediate between the current spacing and your 'compact' spacing, and provide stylesheets with both more-compact and less- compact variants? -Edward |