On 11/28/06, Ueli Kistler <ukistler@hsr.ch> wrote:

Hi, Sorry for the long delay in answering you, but I was traveling and I'm going through my pile of e-mails only now ;-)

short: we need some advice/help on the parser :)
This mail was posted to jython-dev as well.
Please reply if you have some answers to the following questions.

We're working on a Eclipse IDE Python Refactoring plug-in (open-source,
LTK based), see http://sifsstud2.hsr.ch.

Nice to hear about it... what's the license you're using? (Couldn't find it in the homepage)

We're facing 2 main *challenges* using the jython parser (working on
PyDev's jython parser provided for Python 2.4/2.5 grammars):
1. BeginColumn provided are wrong sometimes (we really need correct
values here for correct indentation)

2. Visitor pattern is not applied for all nodes as expected (sometimes
nodes just get traversed.. unnoticed by the visitor)

.. less important:
3. Num-Node didn't provide enough information (fixed by providing numType).

4. Comments may disappear

We would really appreciate some advise/help on this issue!
ClassDef/FunctionDef e.g. have correct beginColumn values... but e.g.
Assign-Nodes usually not.
Can someone check this behaviour and confirm ... give some advice how to
fix it or even provide a fix?

I wrote an AST rewriter visitor (AST -> source code, works quite well)
and noticed that *not* all AST nodes are "properly" handled by the
visitor pattern.

1. Is there any reason why some nodes shouldn't be noticed by the
visitor (accept(VisitorIF visitor) just traversing its "child" nodes,
not calling any visit...-method provided)
2. We would like to contribute our change proposals later ... is the
Jython/PyDev project working together? Where should be post change

Pydev and Jython are not really working together on it... but I think that the jython guys will take a look at the pydev parser when they do come to version 2.4 / 2.5 of python... the main reason why it is not actually the same is because I need much more info than the jython guys, so, overall, I'd be adding much more complexity into the parser than the jython guys need (like getting comments and correct begin columns for instance)

But on the other hand, the parser I have is pretty similar to the parser you have (as I was aiming on doing pretty printing too), and I think that change proposals could go into the pydev feature tracker... and patches are also welcome).

Currently certain nodes are handled differently, but it would be helpful
if the visitor would work like expected.
Please visit
for a more detailed description.

Humm, for this one the solution I'd aim for is creating a new base visitor that you can use in your classes (this is much, much  easier than tweaking the grammar). Although, currently, for the cases I have, I've settled to the current design, as I usually don't have to say transverse an aliasType without knowing the Import info anyway (and the other cases are pretty similar to that I believe... mostly, in those cases it is kind of awkward having the node by itself, so, you usually will have to keep the parent info for those nodes anyway, so, it's just more work for the person writing the visitor... but I can understand that you may want it conceptually different from that).

see http://sifsstud2.hsr.ch/peptic/wiki/Parser

Seems you've already solved those ;-)

see http://sifsstud2.hsr.ch/peptic/wiki/Parser

Not sure why you loose the comments, I'll have to take a look at it.