[Jython-dev] new static analyzer from Google

[Steve Y. <st...@go...>]

I've got a patch to submit for everyone's consideration.  It's a static
analyzer and indexer for Python
source, built atop the Jython antlr parser.  It was primarily designed to
produce an index that can
support IDE functionality, but it could potentially also be used to provide
type information to the
interpreter for optimization purposes.

Some notes about the indexer:

  * it was an intern project from last summer, written from scratch by our
intern Yin Wang
     - I've been tinkering with it for 4 or 5 months, getting it to index
Google's entire Python code base

  * the foundation is a traditional bottom-up/top-down type-inferencer.  Yin
did some really great work here.
    It could be better (what inferencer couldn't?), but it does a nice job
in the absence of type annotations.
    - Jim Baker's suggestion for using decorators for annotations seems like
a great way to feed it type hints
    - I'm avoiding going that route until the Python community gains
consensus around standardized type hints

  * it builds an index of uniquely qualified global names, prefixing them
with package paths as necessary.
    This makes it possible to build a merged index across many projects
without fear of name collisions.

  * it statically models a large portion of the standard library -- some 50
modules and perhaps 1000 definitions.
     - this would be easier and more effective if Python had a way to
annotate return types, but it works OK.

  * it provides some error/warning lint diagnostics, although we haven't
done too much here yet.
    It probably has enough information today to replicate the diagnostics
from every other Python lint tool.

  * it has a decent unit-test suite, with around 500 assertions.  could be
better, but it's a good start.

  * it comes with a quick-and-dirty demo app I wrote last week.  You point
the app at a file or directory,
    and it recursively analyzes/indexes all the source and creates static
html pages with LXR-like links
    between identifiers, per-file structural outlines, and syntax and (some)
semantic highlighting.
    - the demo app doesn't show off some of the fancier features (e.g.
inferred types, xrefs, diagnostics)

There are bugs, of course, and there are many ways it can be improved.  But
it works pretty well.
If nothing else, it's one of the most ass-kicking intern projects I've seen
in the last twenty years.

The indexer is mostly self-contained, residing in a new package
org.python.indexer.  We had
to make some tweaks to Python.g and the support classes, but we didn't
modify the AST; instead
Yin designed a new AST format for the indexer, and wrote a conversion pass
before the analysis.
(We could use the current AST, but we wanted to minimize the diffs on the
existing code for now.)

Yin is back in his Ph.D. program, but he has expressed interest in
continuing work on the analyzer.
My team at Google also plans to continue work on it on a part-time basis, as
it's an important part
of a larger project we're doing in this space.

I'd like to submit a patch for review .  I'm happy to make changes as
needed, provided they're
not earth-shatteringly large feature requests.

The patch is in two parts:  files we needed to change in Jython, and a zip
of new files and directories.
I can't easily put them all in one patch because I can't svn add them to my
working copy of head.

If someone creates a bug or feature request at bugs.jython.org, I should be
able to attach the
patch to it.  Or I can email it directly to one of the maintainers and they
can do it; either way.

Let me know how you'd like to proceed.

Thanks,

-steve