I'd suggest deriving this from our Antlr-based parser, since Antlr has ostensibly the best error recovery support for generated parsers. Frank Wierzbicki knows much more about this, but in looking at Python.g, we just use this error recovery to better match CPython's syntax errors.

But Antlr can do much more, specifically around add/delete a single token to support resynchronization.
Terrence Parr explains his rationale a bit here: http://www.antlr2.org/blog/antlr3/error.handling.tml

NetBeans support for Python (CPython/Jython) uses our Antlr grammar, so there may be some ideas there you can borrow.

Good luck, this sounds like a fun project!

- Jim 

On Sat, May 30, 2009 at 11:08 PM, Yin Wang <yinwang0@gmail.com> wrote:
Thanks a lot! My original purpose is to have a fault-tolerant parser
to use in an IDE-like environment where code can be checked to the end
even syntax errors exist, so I'd like to have a parser that has some
recovery abilities. Could you suggest some parsers that can do that? I
know Pydev's parsers can recover from syntax errors but they seem to
be tightly connected with eclipse. Thanks.

-- yin

On Fri, May 29, 2009 at 4:50 PM, Jim Baker <jbaker@zyasoft.com> wrote:
> Yin,
> PythonPartial is not meant to return an AST. Its sole purpose is to
> determine if the input presented at the console is complete or not. For the
> Python REPL, we need a separate grammar because it's OK if the partial parse
> contains a fragment, such as a multiline string that's not been terminated
> with a """.
> PythonPartial is generated from PythonPartial.g, exposed by ParserFacade,
> and documented in terms of its usage in the comments for
> InteractiveInterpreter#runsource.
> If you compare PythonPartial.g and Python.g, you will see the specifics of
> this (I would not really look at the generated source in
> PythonPartial.java). PythonPartial.g is just a parser grammar, with some
> sharing against Python.g for tokens (tokenVocab). There is no AST
> construction performed. In contrast, Python.g contains the tree construction
> as needed for subsequent compilation, and is consequently a much larger
> file.
> - Jim
> On Sat, May 30, 2009 at 4:28 AM, Yin Wang <yinwang0@gmail.com> wrote:
>> Hi,
>> I'm trying to figure out how to use the partial parser
>> org.python.antlr.PythonPartial, but I haven't found how to get an AST
>> from it. It doesn't even refer to any type in the PythonTree
>> hierarchy. Strange. Any ideas or code examples? The file
>> org.python.antlr.PythonPartialTester simple succeeds without giving
>> out any value. It calls single_input() but that method's return type
>> is void. Thanks.
>> -- yin
>> ------------------------------------------------------------------------------
>> Register Now for Creativity and Technology (CaT), June 3rd, NYC. CaT
>> is a gathering of tech-side developers & brand creativity professionals.
>> Meet
>> the minds behind Google Creative Lab, Visual Complexity, Processing, &
>> iPhoneDevCamp as they present alongside digital heavyweights like
>> Barbarian
>> Group, R/GA, & Big Spaceship. http://p.sf.net/sfu/creativitycat-com
>> _______________________________________________
>> Jython-dev mailing list
>> Jython-dev@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/jython-dev
> --
> Jim Baker
> jbaker@zyasoft.com

Jim Baker