I've got a patch to submit for everyone's consideration. It's a static analyzer and indexer for Python
source, built atop the Jython antlr parser. It was primarily designed to produce an index that can
support IDE functionality, but it could potentially also be used to provide type information to the
interpreter for optimization purposes.
Some notes about the indexer:
* it was an intern project from last summer, written from scratch by our intern Yin Wang
- I've been tinkering with it for 4 or 5 months, getting it to index Google's entire Python code base
* the foundation is a traditional bottom-up/top-down type-inferencer. Yin did some really great work here.
It could be better (what inferencer couldn't?), but it does a nice job in the absence of type annotations.
- Jim Baker's suggestion for using decorators for annotations seems like a great way to feed it type hints
- I'm avoiding going that route until the Python community gains consensus around standardized type hints
* it builds an index of uniquely qualified global names, prefixing them with package paths as necessary.
This makes it possible to build a merged index across many projects without fear of name collisions.
* it statically models a large portion of the standard library -- some 50 modules and perhaps 1000 definitions.
- this would be easier and more effective if Python had a way to annotate return types, but it works OK.
* it provides some error/warning lint diagnostics, although we haven't done too much here yet.
It probably has enough information today to replicate the diagnostics from every other Python lint tool.
* it has a decent unit-test suite, with around 500 assertions. could be better, but it's a good start.
* it comes with a quick-and-dirty demo app I wrote last week. You point the app at a file or directory,
and it recursively analyzes/indexes all the source and creates static html pages with LXR-like links
between identifiers, per-file structural outlines, and syntax and (some) semantic highlighting.
- the demo app doesn't show off some of the fancier features (e.g. inferred types, xrefs, diagnostics)
There are bugs, of course, and there are many ways it can be improved. But it works pretty well.
If nothing else, it's one of the most ass-kicking intern projects I've seen in the last twenty years.
The indexer is mostly self-contained, residing in a new package org.python.indexer. We had
to make some tweaks to Python.g and the support classes, but we didn't modify the AST; instead
Yin designed a new AST format for the indexer, and wrote a conversion pass before the analysis.
(We could use the current AST, but we wanted to minimize the diffs on the existing code for now.)
Yin is back in his Ph.D. program, but he has expressed interest in continuing work on the analyzer.
My team at Google also plans to continue work on it on a part-time basis, as it's an important part
of a larger project we're doing in this space.
I'd like to submit a patch for review . I'm happy to make changes as needed, provided they're
not earth-shatteringly large feature requests.
The patch is in two parts: files we needed to change in Jython, and a zip of new files and directories.
I can't easily put them all in one patch because I can't svn add them to my working copy of head.
If someone creates a bug or feature request at bugs.jython.org
, I should be able to attach the
patch to it. Or I can email it directly to one of the maintainers and they can do it; either way.
Let me know how you'd like to proceed.