Thread: [Pyparsing] DSL using pyparsing
Brought to you by:
ptmcg
From: Asim M. <as...@ho...> - 2009-06-10 18:57:48
|
Ok i am looking at building a DSL using pyparsing and am relatively new to python . Intially I am looking at something like this: A = {build some list, this calls some underlying python api and creates say a class A} B = {build another list, this creates another python class B } C= A+B Should i parse this as a single script or parse each statement? If i parse each statement how can i pass variables that were defined in the previous statement? As the complexity of the doamin increase, it might be the case that the construction of A depends on other statements. What i am looking for is the best approach. Thanks |
From: Paul M. <pt...@au...> - 2009-06-10 19:17:54
|
Asim - I would work on this like they dig a tunnel - from both directions. First I would write out what my DSL syntax would be. Then I would work backwards and write out how this would be implemented in Python. The purpose of the pyparsing DSL converter is to find your DSL syntax, and then create the related Python code. Let's say your application was something about RGB colors, and A and B are going to be colors which, when added together, result in a third color whose RGB values are the respective sums of the R, G, and B values of A and B. Let's say this is your color syntax: A = < 100 10 50 > B = < 0 0 0 > C = A+B The purpose of the DSL converter is to convert the "<>" syntax into a Python-compatible API call (probably a constructor call to a Color class). The "C=A+B" line probably doesn't need DSL handling, your Python Color class can define the __add__() function to do the proper R, G, and B summing. So all your DSL converter in this case would need to do would be to change: A = < 100 10 50 > To: A = Color(100, 10, 50) In pyparsing this is a very simple thing: LT,GT = map(Suppress,"<>") Integer = Word(nums) colorDef = LT + integer*3 + RT colorDef.setParseAction(lambda tokens : "Color(%s, %s, %s)" % tokens) Then use colorDef.transformString() to convert the source read from the imported file into valid Python, and then use that to compile into the executable module. Remember, you are generating a Python script from your DSL script, so you don't have to worry about "carrying values forward from one line to the next" - when your generated code gets compiled to Python bytecode, it will do this for you. Don't try to re-invent Python with your DSL. You are *much* better off using your DSL to *augment* the Python syntax. Thanks for reading the article! -- Paul > -----Original Message----- > From: Asim Malik [mailto:as...@ho...] > Sent: Wednesday, June 10, 2009 1:58 PM > To: pyp...@li... > Subject: [Pyparsing] DSL using pyparsing > > > Ok i am looking at building a DSL using pyparsing and am relatively new to > python . Intially I am looking at something like this: > > A = {build some list, this calls some underlying python api and creates > say a class A} > B = {build another list, this creates another python class B } > C= A+B > > Should i parse this as a single script or parse each statement? If i parse > each statement how can i pass variables that were defined in the previous > statement? As the complexity of the doamin increase, it might be the case > that the construction of A depends on other statements. What i am looking > for is the best approach. > > Thanks > > > > -------------------------------------------------------------------------- > ---- > Crystal Reports - New Free Runtime and 30 Day Trial > Check out the new simplified licensing option that enables unlimited > royalty-free distribution of the report engine for externally facing > server and web deployment. > http://p.sf.net/sfu/businessobjects > _______________________________________________ > Pyparsing-users mailing list > Pyp...@li... > https://lists.sourceforge.net/lists/listinfo/pyparsing-users |
From: Asim M. <as...@ho...> - 2009-06-12 00:49:05
|
Thanks Paul, I think i have the idea now. I am using Imputil, so the problem i have now is that i am using imputil, so when say i am trying to compile a script, say: A = Color(100, 10, 50) I get a "NameError: name 'Color' is not defined". Any ideas? > From: pt...@au... > To: as...@ho...; pyp...@li... > Subject: RE: [Pyparsing] DSL using pyparsing > Date: Wed, 10 Jun 2009 14:17:50 -0500 > > Asim - > > I would work on this like they dig a tunnel - from both directions. First I > would write out what my DSL syntax would be. Then I would work backwards > and write out how this would be implemented in Python. The purpose of the > pyparsing DSL converter is to find your DSL syntax, and then create the > related Python code. > > Let's say your application was something about RGB colors, and A and B are > going to be colors which, when added together, result in a third color whose > RGB values are the respective sums of the R, G, and B values of A and B. > Let's say this is your color syntax: > > A = < 100 10 50 > > B = < 0 0 0 > > C = A+B > > The purpose of the DSL converter is to convert the "<>" syntax into a > Python-compatible API call (probably a constructor call to a Color class). > The "C=A+B" line probably doesn't need DSL handling, your Python Color class > can define the __add__() function to do the proper R, G, and B summing. > > So all your DSL converter in this case would need to do would be to change: > > A = < 100 10 50 > > > To: > > A = Color(100, 10, 50) > > In pyparsing this is a very simple thing: > > LT,GT = map(Suppress,"<>") > Integer = Word(nums) > colorDef = LT + integer*3 + RT > colorDef.setParseAction(lambda tokens : "Color(%s, %s, %s)" % tokens) > > Then use colorDef.transformString() to convert the source read from the > imported file into valid Python, and then use that to compile into the > executable module. > > Remember, you are generating a Python script from your DSL script, so you > don't have to worry about "carrying values forward from one line to the > next" - when your generated code gets compiled to Python bytecode, it will > do this for you. > > Don't try to re-invent Python with your DSL. You are *much* better off > using your DSL to *augment* the Python syntax. > > Thanks for reading the article! > > -- Paul > > > > -----Original Message----- > > From: Asim Malik [mailto:as...@ho...] > > Sent: Wednesday, June 10, 2009 1:58 PM > > To: pyp...@li... > > Subject: [Pyparsing] DSL using pyparsing > > > > > > Ok i am looking at building a DSL using pyparsing and am relatively new to > > python . Intially I am looking at something like this: > > > > A = {build some list, this calls some underlying python api and creates > > say a class A} > > B = {build another list, this creates another python class B } > > C= A+B > > > > Should i parse this as a single script or parse each statement? If i parse > > each statement how can i pass variables that were defined in the previous > > statement? As the complexity of the doamin increase, it might be the case > > that the construction of A depends on other statements. What i am looking > > for is the best approach. > > > > Thanks > > > > > > > > -------------------------------------------------------------------------- > > ---- > > Crystal Reports - New Free Runtime and 30 Day Trial > > Check out the new simplified licensing option that enables unlimited > > royalty-free distribution of the report engine for externally facing > > server and web deployment. > > http://p.sf.net/sfu/businessobjects > > _______________________________________________ > > Pyparsing-users mailing list > > Pyp...@li... > > https://lists.sourceforge.net/lists/listinfo/pyparsing-users > |
From: Paul M. <pt...@au...> - 2009-06-12 02:11:58
|
> Thanks Paul, I think i have the idea now. I am using Imputil, so the > problem i have now is that i am using imputil, so when say i am trying to > compile a script, say: > > A = Color(100, 10, 50) > > I get a "NameError: name 'Color' is not defined". Any ideas? > > Well, you *are* generating Python code, so if you had originally written that line of code you would get the same error. The Color class needs to come from somewhere. Either import it from another module, or define it somewhere in your source program (maybe in the module you import to include the imputil hook). -- Paul |
From: Asim M. <as...@ho...> - 2009-06-12 11:31:29
|
So is there a way for the import hook to somehow inject the import statements, when executing the code? Or does my generated code have to explicitly declare each import statement? Let me try an break it down, suppose i have this code in a file 'Colors.py', the path of which is (xxx.dsl.Colors.py) and for now the Color class resides here but it could be imported from some other external module: import imputil class Color(object): def __init__(self, r, g, b): self.r = r self.g = g self.b = b def __str__(self): return 'cols' def convert_dsl(filepath, fileinfo, filename): srcfile = open(filepath, 'r') data = srcfile.read() srcfile.close() generated_code = '%s' % data print generated_code COMPILE_MODE = 'exec' codeobj = compile(generated_code.rstrip(" \t"), filepath, COMPILE_MODE) return 0, codeobj, {} importer = imputil.ImportManager() importer.add_suffix('.col', convert_dsl) importer.install() --- Is this all that needs to be done or do i have to add some namespace or something? Then i have my abc.col file: c = Color(255, 0, 255) In reality this would be a file which has the dsl that would be py-parsed, so obviously i don't want an import statement in this. If i did add 'from Colors import Color' at the top it works, but in reality i am expecting 'from xxx.dsl.Colors import Color' Now if i run this, i get: from xxx.dsl.Colors import * import abc print abc.c I get the error: c = Color(255, 0, 255) NameError: name 'Color' is not defined. I am not quite sure where the imports should be defined or resolved from. > From: pt...@au... > To: as...@ho...; pyp...@li... > Subject: RE: [Pyparsing] DSL using pyparsing > Date: Thu, 11 Jun 2009 21:11:56 -0500 > > > Thanks Paul, I think i have the idea now. I am using Imputil, so the > > problem i have now is that i am using imputil, so when say i am trying to > > compile a script, say: > > > > A = Color(100, 10, 50) > > > > I get a "NameError: name 'Color' is not defined". Any ideas? > > > > > > Well, you *are* generating Python code, so if you had originally written > that line of code you would get the same error. The Color class needs to > come from somewhere. Either import it from another module, or define it > somewhere in your source program (maybe in the module you import to include > the imputil hook). > > -- Paul > > |
From: Paul M. <pt...@au...> - 2009-06-12 14:19:05
|
> -----Original Message----- > From: Asim Malik [mailto:as...@ho...] > Sent: Friday, June 12, 2009 6:31 AM > To: pt...@au...; pyp...@li... > Subject: RE: [Pyparsing] DSL using pyparsing > > So is there a way for the import hook to somehow inject the import > statements, when executing the code? Or does my generated code have to > explicitly declare each import statement? > Have your import hook inject the necessary import statements at the top of the generated code: generated_code = """from Color import Color %s""" % data Here I put your Color class in a file named Color.py. Now when I run the script testColor.py, which contains this code: import Colors import abc print abc.c I get: from Color import Color c = Color(255, 0, 255) cols Now the last step is for you to convert your assignment to generated_code to use a pyparsing expression, and call transformString on the content of the file (in your variable 'data'). -- Paul |