The token format is described in the `semantic-token-style-guide'.
As new parts were added, I've tried keeping it up to date.
Of course, we had plans for a while to switch to a different token
format. Don't know when we will get to that.
>>> ryk@... seems to think that:
>Thanks for your help. I checked in your changes pretty much
>as is. So simple formal parameters are now parsed and
>included within the 'function token. Other changes checked
>in include clean up of many semantic actions, as well as
>parsing of base class list within 'class tokens.
>I also checked couple of sentences in NEWS files.
>Next I'll be studying the tokens generated by the java
>parser and I'll try to have the python parser generate
>similar tokens. In particular, I'll try to include
>auxiliary informations such as formal parameters and base
>class lists in the same position within the token list.
>After that, I will go through as many files as I can in
>/usr/lib/python2.2/ directory and make sure that the parser
>can handle all the files.
>>>>>> "DP" == David PONCE <David.Ponce@...> writes:
> DP> Hi Richard,
> >> I would like to ask your help in figuring out why python grammar does
> >> not seem to work as I expected. I include the full copy of
> >> wisent-python.wy at the end, but here are the three NT's of interest.
> DP> [...]
> >> This is my attempt at ading function arguments to 'function tokens.
> >> As you can see the code above is very similar to what you have in
> >> wisent-java-tag.wy. The last NT, function_parameters, is very
> >> simplified version since it only matches NAME terminal.
> DP> [...]
> >> The problem I can't solve now is that if I add
> >> %start function_parameters
> >> then rebuild the tables and re-bovinate, then no token is generated
> >> from the same input lines! It seems like adding this one line
> >> seems to break the parser. What is going on?
> DP> If fact the result of an EXPANDFULL clause nonterminal must be a
> DP> valid semantic token otherwise the iterative parser just return nil.
> DP> As your NAME rule just return a string (value of NAME) it is not
> DP> considered as a valid token and ignored.
> DP> After some minor changes in your grammar I managed to successfully
> DP> parse your small example below:
> DP> def ttt(x):
> DP> i = 1
> DP> That produced this parse tree:
> DP> (("ttt" function nil
> DP> (("x" variable nil nil nil nil
> DP> ((reparse-symbol . function_parameters))
> DP> #<overlay from 9 to 10 in test.py>))
> DP> nil #<overlay from 1 to 23 in test.py>))
> DP> I attached the new grammar. Using diff you will easily see what has
> DP> changed. Particularly I had to remove the NEWLINE following
> DP> compound_stmt in the goal nonterminal, as it appeared to me that the
> DP> lexical stream never terminates with a NEWLINE lexical token!
> DP> Hope it helps.
> DP> Thanks for working on that!
> DP> David
>This SF.NET email is sponsored by:
>SourceForge Enterprise Edition + IBM + LinuxWorld = Something 2 See!
>Cedet-devel mailing list
Eric Ludlam: zappo@..., eric@...
Home: http://www.ludlam.net Siege: http://www.siege-engine.com
Emacs: http://cedet.sourceforge.net GNU: http://www.gnu.org