Thread: [Pyparsing] A PyParsing hack
Brought to you by:
ptmcg
From: Gre7g L. <haf...@ya...> - 2008-03-07 19:36:15
|
First off, as a new user of PyParsing, I've gotta' say that I am hooked. This thing rocks and is better than sliced bread in a variety of ways. Next, it does almost everything I could ever want... but because it's open source, I'm able to make changes to make it suit me. How cool is that? It's like every day is my birthday, or something. So, what follows is not really a question. I just wanted to share what I'm doing and to make this available to anyone else who might need to do something similar. I guess if there is any element of question to this e-mail, it would be, "Do you think this is the cleanest way to add this functionality in (assuming that it wasn't already here and I missed it somehow)?" and if so, "How can I contribute this back to the project?" On to the show... As I mentioned yesterday, I'm using PyParsing to parse user code into something that my hardware can run. I want to give the user a stripped-down version of C, but due to extreme space limitations in my hardware, I can't use any pre-existing virtualization code. So let's say the user entered the function: X=Y+5*Z I would want to create instances that would represent the various pieces of the instruction (one for each variable, one for the multiplication, one for the addition, and one for the assignment) and then call some recursive member of the assignment instance to generate the op-codes required to load/store and calculate. Teaching PyParsing to parse the assignment is pretty trivial. With minimal effort, I can get it to churn out the hierarchial list: [['X', '=', ['Y', '+', [5, '*', 'Z']]]] At this point, I *COULD* traverse back through the output and convert the various pieces to instances, but in essence, I would be doing the parsing process twice. What I really would like is for PyParsing to use its slick-as-snot setParseAction function to name the class for me so I can just sit back and enjoy. Here's a patch you can apply to the SVN head to make setParseAction accept classes in addition to functions: http://pastie.textmate.org/162921 It only really does two things: 1. It accepts the class in lieu of a function, and 2. It provides a base class that the user can derive from to prevent a reduce-to-text operation during the parsing process. With this code in place, the assignment is now returned as (formatting added manually): [ <Assignment <Token 'X'> = <Math [<Token 'Y'>, '+', <Math [5, '*', <Token 'Z'>]> ] > > ] And finally, here's a really simple program to do the above: http://pastie.textmate.org/162922 My actual code handles a lot more kinds of code structures and is still being debugged, but I didn't want to bog everyone down in that, hence the tiny snippet. Finally, I tried to run the unitTests.py file to verify that I hadn't broken anything in the process, but I don't see all the required files in SVN. Thanks again, Gre7g __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com |
From: Paul M. <pa...@al...> - 2008-03-07 20:27:20
|
Greg - Bwa-ha-ha! Another minion in my evil domain!!! :) I'm very happy that you find pyparsing so useful AND enjoyable. Your concept of passing a class instead of a function to setParseAction looks pretty good. Your point is dead-on - why bother returning just a nested list of strings when you can use parse actions to construct and return executable objects? It is an excellent combination of both lexing and parsing. And extra credit for using parse actions as part of an operatorPrecedence expression. Please don't be disappointed, but yes, pyparsing *can* do this already. In fact, the code you posted/pasted, if you just change PP.NoStringBase to object as the base class for your various parse action/classes, will run otherwise unchanged with the unpatched pyparsing running on my system. I get this result: [<Assignment <Token 'X'> = <Math [<Token 'Y'>, '+', <Math [5, '*', <Token 'Z'>] >] > >] For another example of this technique, see: http://pyparsing.wikispaces.com/space/showimage/simpleBool.py. In addition to being online at the pyparsing wiki, this script is included in the examples directory of the source and doc distibutions. In "Getting Started With Pyparsing" (http://www.oreilly.com/catalog/9780596514235), the final chapter shows how to convert query string expressions into objects that generate Python set expressions, which can then be intersect'ed and union'ed to correspond to the and's and or's of the original query string. And I have a new magazine article coming out soon that uses this method to compile an esoteric language to executable objects. Congratulations on arriving at this powerful concept on your own, it is most definitely an advanced technique! And keep proposing patches! There are now many more brains than just my own thinking about this module and how it could be improved, and I would be foolish to limit new work to only the stuff I come up with myself. Many of the recent enhancements came as a result of discussions and patches submitted over discussion lists and forums. I'm glad that you are not afraid to get your fingernails dirty, and get right into the pyparsing code to see what it is doing. -- Paul (I'm sorry about unitTests - some of the files that I parse in my testing are proprietary to some of my business customers, so I can't post them. I'll try to update the SVN with what is publicly shareable, and conditionalize the unitTests.py script to step over the private tests.) |