pyparsing-users — User notes and help on the pyparsing module

Re: [Pyparsing] pyparsing bug or operator error with cStyleComment?

[spir <den...@fr...>]

Le Mon, 27 Jul 2009 23:05:59 -0400 (EDT),
Paul Wouters <pa...@xe...> s'exprima ainsi:

> 
> Hi,
> 
> I'm trying to build a better working parser for named.conf files, and ran
> into a strange issue while parsing various kinds of comments. I think the
> below examples shows what I think is a bug:
> 
> from pyparsing import *
> 
> stuff = """
> /* comment 1*/
> 
> // test
> // test2
> /* test3
>     indeed */
> /* test4
> */
> """
> 
> comment1 = Group(Word("//") + restOfLine)
> comment2 = Group(Word("#") + restOfLine)
> comment3 = cStyleComment
> comment = OneOrMore(comment1 | comment2 | comment3)
> parser = OneOrMore(comment)
> 
> res = parser.parseString(stuff,parseAll=True)
> 
> This gives me:
> 
> pyparsing.ParseException: Expected end of text (at char 46), (line:7, col:4)
> I think that "cStyleComment" comments do not correctly work over multiple
> lines.
> 
> Paul

well, you're wrong about cStyleComment, at least:

===========
from pyparsing import cStyleComment

stuff = """/* test
	   test   	
test */"""

print cStyleComment.parseString(stuff)
# ==> '/* test\n           test         \ntest */'
===========

Also, comment's format should be simply "comment = comment1 | comment2 | comment3".


See also below:
===========
from pyparsing import cStyleComment,OneOrMore

stuff = """
/* test3
    indeed */
/* test4
*/"""

print OneOrMore(cStyleComment).parseString(stuff)
# ==> ['/* test3\n    indeed */', '/* test4\n*/']
===========

This said, I couldn't find the reason of the bug.

Denis
------
la vita e estrany

[Pyparsing] pyparsing bug or operator error with cStyleComment?

[Paul W. <pa...@xe...>]

Hi,

I'm trying to build a better working parser for named.conf files, and ran
into a strange issue while parsing various kinds of comments. I think the
below examples shows what I think is a bug:

from pyparsing import *

stuff = """
/* comment 1*/

// test
// test2
/* test3
    indeed */
/* test4
*/
"""

comment1 = Group(Word("//") + restOfLine)
comment2 = Group(Word("#") + restOfLine)
comment3 = cStyleComment
comment = OneOrMore(comment1 | comment2 | comment3)
parser = OneOrMore(comment)

res = parser.parseString(stuff,parseAll=True)

This gives me:

pyparsing.ParseException: Expected end of text (at char 46), (line:7, col:4)

I think that "cStyleComment" comments do not correctly work over multiple
lines.

Paul

[Pyparsing] pyparsing bug or operator error with cStyleComment?

[Paul W. <pa...@xe...>]

Hi,

I'm trying to build a better working parser for named.conf files, and ran
into a strange issue while parsing various kinds of comments. I think the
below examples shows what I think is a bug:

from pyparsing import *

stuff = """
/* comment 1*/

// test
// test2
/* test3
    indeed */
/* test4
*/
"""

comment1 = Group(Word("//") + restOfLine)
comment2 = Group(Word("#") + restOfLine)
comment3 = cStyleComment
comment = OneOrMore(comment1 | comment2 | comment3)
parser = OneOrMore(comment)

res = parser.parseString(stuff,parseAll=True)

This gives me:

pyparsing.ParseException: Expected end of text (at char 46), (line:7, col:4)
I think that "cStyleComment" comments do not correctly work over multiple
lines.

Paul

[Pyparsing] Announcing Booleano

[Gustavo N. <me...@gu...>]

Hello, everyone.

I am proud to announce the first alpha release of Booleano, an interpreter of 
boolean expressions, a library to define and run filters available as text 
(e.g., in a natural language) or in Python code.

In order to handle text-based filters, Booleano ships with a fully-featured 
parser whose grammar is adaptive: Its properties can be overridden using 
simple configuration directives. This parser is powered by Pyparsing.

On the other hand, the library exposes a pythonic API for filters written in 
pure Python. These filters are particularly useful to build reusable 
conditions from objects provided by a third party library.

It's been designed to address the following use cases:
 1.- Convert text-based conditions: When you need to turn a condition 
available as plain text into something else (i.e., another filter).
 2.- Evaluate text-based conditions: When you have a condition available as 
plain text and need to iterate over items in order to filter out those for 
which the evaluation of the condition is not successful.
 3.- Evaluate Python-based conditions: When you have a condition represented 
by a Python object (nothing to be parsed) and need to iterate over items in 
order to filter out those for which the evaluation of the condition is not 
successful.

This release is absolutely usable, but the documentation is far from complete 
because I needed this release out for another project I need to work on ASAP 
(it will depend on Booleano). The next release will ship with a nice 
documentation, I promise.

Website: http://code.gustavonarea.net/booleano/

I hope you'll like it! Comments are most welcome :)

Cheers,
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] setDefaultWhitespaceChars

[Paul M. <pt...@au...>]

> I am trying to make "\t" significant by passing a string that doesn't
> include "\t" to setDefaultWhitespaceChars.
> However, it still seems to recognize '\t' as a white space.
> 
> >>> ParserElement.setDefaultWhitespaceChars("\n\r ")
> >>> words = Word(alphas)
> >>> sentence = OneOrMore(words) + '.'
> >>> print sentence.parseString("\tHello World.")
> ['Hello', 'World', '.']
> 

This requirement doesn't come up too often, so it takes people by surprise.
In order to keep columns straight, parseString calls expandtabs on the
provided string before it begins parsing.  You can suppress this by calling
the parseWithTabs method on your top-level expression:

>>> sentence.parseWithTabs()
{{W:(abcd...)}... "."}
>>> print sentence.parseString("\tHello World.")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python25\lib\site-packages\pyparsing.py", line 1076, in
parseString
    raise exc
pyparsing.ParseException: Expected W:(abcd...) (at char 0), (line:1, col:1)

-- Paul

[Pyparsing] setDefaultWhitespaceChars

[<mmm...@gm...>]

I am trying to make "\t" significant by passing a string that doesn't  
include "\t" to setDefaultWhitespaceChars.
However, it still seems to recognize '\t' as a white space.

>>> ParserElement.setDefaultWhitespaceChars("\n\r ")

>>> words = Word(alphas)

>>> sentence = OneOrMore(words) + '.'

>>> print sentence.parseString("\tHello World.")

['Hello', 'World', '.']


How should I accomplish this?

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Gustavo N. <me...@gu...>]

Hello, Ralph.

Ralph said:
> `^' causes all alternatives to be tested, `|' stops at the first one
> that succeeds.  If using `|', unless you test for `<=' before `<' then
> `<=' would never be seen because it would always look like a `<'.  With
> `^', I assume the longest one that succeeds is used so both `<' and `<='
> match but the latter is used.

Hmm, OK, I got it now. Thank you very much for the explanation! ;-)

Cheers.
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Ralph C. <ra...@in...>]

hi Gustavo,

> > 2.
> >     relationals = eq ^ ne ^ lt ^ gt ^ le ^ ge
> >
> > '^' can be relatively expensive for pyparsing if there are many
> > alternatives, since all alternatives will be checked.  This version
> > is equivalent:
> >
> >     relationals = eq | ne | le | ge | lt | gt
> 
> Thanks for the hint. I used pipes initially but for some reason some
> tests didn't pass... I've just restored the pipes and the all the
> tests pass now, though :-S
> 
> > (I *did* have to be careful about the order, having to check for
> > "<=" before "<", and ">=" before ">".)
> 
> Could you please elaborate on that? Is it for performance reasons?

`^' causes all alternatives to be tested, `|' stops at the first one
that succeeds.  If using `|', unless you test for `<=' before `<' then
`<=' would never be seen because it would always look like a `<'.  With
`^', I assume the longest one that succeeds is used so both `<' and `<='
match but the latter is used.

Cheers,


Ralph.

Re: [Pyparsing] Is it possible to optimize this grammar?

[Gustavo N. <me...@gu...>]

What!? I'd call this "Mega Turbo Button", not just packrat. ;-)

"W == 0   |   (X != 1   ^   (Y > 2   &   Z < 3))" is parsed without packrat in 
676.20 seconds. With packrat, it's parsed in 0.02 seconds! 33810x faster!

And the best of all is that all my 500+ tests still pass. Now the test suite 
is run in just 1.53 seconds!

This is just awesome. Thank you very much, Paul!

  - Gustavo.


Paul said:
> Parsers that have deep operatorPrecedence definitions often benefit from
> using packrat parsing.  Please insert this at the top of your module, right
> after the import statement (you'll also have to add ParserElement to the
> list of names imported from the pyparsing module):
>
> ParserElement.enablePackrat()
>
>
> Packrat parsing is a kind of parse-time memoization.  There is more info on
> the FAQs page of the wiki.
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Gustavo N. <me...@gu...>]

Hello, Paul!

First of all, thank you very much for your help. :)

Paul said:
> Gustavo -
>
> The first issue you have is your definition of identifier0, which you
> define using the Unicode regex "\w", but which cannot start with a numeric
> digit. Unfortunately, there are a lot more numeric digits in Unicode than
> just "0" through "9" - over 300 of them!  Here is how I replaced your
> identifier0 expression, and got your code to at least start working:
>
>     unidigit = Regex(u"[" + "".join(unichr(c) for c in xrange(0x10000) if
> unichr(c).isdigit()) + "]", re.UNICODE)
>     identifier0 = Combine(~unidigit + Regex("[\w]+", re.UNICODE))
 
Thanks, I've just replaced my solution (which used a parse action) with this.


> Some other comments:
>
> 1.
>
>     eq = CaselessLiteral("==")
>     ne = CaselessLiteral("!=")
>     lt = CaselessLiteral("<")
>     gt = CaselessLiteral(">")
>     le = CaselessLiteral("<=")
>     ge = CaselessLiteral(">=")
>
> Why are these not just plain Literal's?  Making them caseless just makes
> pyparsing do extra work trying to match upper or lower case "==".

Sorry, I forgot to explain that: In the actual code, I don't use those 
literals; I use variables because those tokens can be overridden by the user 
(e.g., the equality token could be "is" instead of "=="), so I want those 
tokens to be case-insensitive if applicable.


> 2.
>     relationals = eq ^ ne ^ lt ^ gt ^ le ^ ge
>
> '^' can be relatively expensive for pyparsing if there are many
> alternatives, since all alternatives will be checked.  This version is
> equivalent:
>
>     relationals = eq | ne | le | ge | lt | gt

Thanks for the hint. I used pipes initially but for some reason some tests 
didn't pass... I've just restored the pipes and the all the tests pass now, 
though :-S


> (I *did* have to be careful about the order, having to check for "<="
> before "<", and ">=" before ">".)

Could you please elaborate on that? Is it for performance reasons?


> 3.
>     not_ = Suppress("~")
>     and_ = Suppress("&")
>     in_or = Suppress("|")
>     ex_or = Suppress("^")
>
> Do you really want to suppress these operators?  Without them, you will
> have a dickens of a time evaluating the parsed expression.  These should
> probably be Literal's, too.

I think so: In the actual code, I have my own parse nodes (for operations and 
operands), and I use the parse actions to convert Pyparsing's parse trees into 
my own parse tree.

For example, the parse action for and_ looks like this:
    def make_and(tokens):
        left_operand = tokens[0][0]
        right_operand = tokens[0][1]
        return BooleanoAnd(left_operand, right_operand)

So the operator isn't necessary to make the new parse tree.


> 4. The parsed results show an empty [] before every identifier, I think
> this is to represent the empty leading namespace for each.  You might want
> to try this for identifier:
>
>     identifier = Combine(namespace.setResultsName("namespace_parts") +
>                   identifier0.setResultsName("identifier"))
>
> Combine will return the match as a single string, but you can still access
> the individual parts of the identifier by their results names if you need
> to.  (Combine will also ensure that you match only contiguous characters as
> an identifier.)

Nice, I've just updated my code accordingly.


> Looks interesting, keep us posted.

Sure! I'm getting closer to the first alpha release, by the way (for the 
impatient: https://launchpad.net/booleano) ;-)

Cheers!
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Is it possible to optimize this grammar?

[Paul M. <pt...@au...>]

Gustavo -

Parsers that have deep operatorPrecedence definitions often benefit from
using packrat parsing.  Please insert this at the top of your module, right
after the import statement (you'll also have to add ParserElement to the
list of names imported from the pyparsing module):

ParserElement.enablePackrat()


Packrat parsing is a kind of parse-time memoization.  There is more info on
the FAQs page of the wiki.

-- Paul

[Pyparsing] Is it possible to optimize this grammar?

[Gustavo N. <me...@gu...>]

Hello, everyone.

My Pyparsing grammar for boolean expressions [1] takes too much time to parse 
relatively complex expressions and I was wondering if it could be optimized 
somehow.

With it, my test suite takes ~1 hour to run, while the following individual 
expressions take the time specified on the right (in seconds):
 * 'user == "gustavo"' => 0.08
 * "weather:today_rains & (Pi > e | today_is_monday())" => 9.78
 * "W == 0   |   (X != 1   ^   (Y > 2   &   Z < 3))" => 676.20

Possibly the worst of all is that these are the results on a 64-bit, 2.50GHz, 
4GiB/RAM computer. So it will be many times slower on many computers. :(

Is there any way I could improve the performance of this parser?

Thanks in advance,

[1] I've excerpted the relevant code here (it's a simplified version of the 
actual code):
http://pyparsing.pastebin.com/f6a9c86ee
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Paul M. <pt...@au...>]

Gustavo -

The first issue you have is your definition of identifier0, which you define
using the Unicode regex "\w", but which cannot start with a numeric digit.
Unfortunately, there are a lot more numeric digits in Unicode than just "0"
through "9" - over 300 of them!  Here is how I replaced your identifier0
expression, and got your code to at least start working:

    unidigit = Regex(u"[" + "".join(unichr(c) for c in xrange(0x10000) if
unichr(c).isdigit()) + "]", re.UNICODE)
    identifier0 = Combine(~unidigit + Regex("[\w]+", re.UNICODE))


Some other comments:

1. 

    eq = CaselessLiteral("==")
    ne = CaselessLiteral("!=")
    lt = CaselessLiteral("<")
    gt = CaselessLiteral(">")
    le = CaselessLiteral("<=")
    ge = CaselessLiteral(">=")

Why are these not just plain Literal's?  Making them caseless just makes
pyparsing do extra work trying to match upper or lower case "==".


2. 
    relationals = eq ^ ne ^ lt ^ gt ^ le ^ ge

'^' can be relatively expensive for pyparsing if there are many
alternatives, since all alternatives will be checked.  This version is
equivalent:

    relationals = eq | ne | le | ge | lt | gt

(I *did* have to be careful about the order, having to check for "<=" before
"<", and ">=" before ">".)


3. 
    not_ = Suppress("~")
    and_ = Suppress("&")
    in_or = Suppress("|")
    ex_or = Suppress("^")

Do you really want to suppress these operators?  Without them, you will have
a dickens of a time evaluating the parsed expression.  These should probably
be Literal's, too.


4. The parsed results show an empty [] before every identifier, I think this
is to represent the empty leading namespace for each.  You might want to try
this for identifier:

    identifier = Combine(namespace.setResultsName("namespace_parts") +
                  identifier0.setResultsName("identifier"))

Combine will return the match as a single string, but you can still access
the individual parts of the identifier by their results names if you need
to.  (Combine will also ensure that you match only contiguous characters as
an identifier.)


Looks interesting, keep us posted.

-- Paul

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Gustavo N. <me...@gu...>]

Hello, Paul!

Paul said:
> I'm afraid your attachments didn't make it through.  Could you paste your
> code extract to http://pyparsing.pastebin.com and send us the link?

I'm sorry about that. Here it is:
http://pyparsing.pastebin.com/f7fa3659d

Thanks in advance,
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Indefinitely recursive grammar because of unaryoperation in operatorPrecedence()

[Gustavo N. <me...@gu...>]

Hello again, Paul!

Paul said:
> Gustavo -
>
> Once again, your attachments are not coming through the list server, so I
> suggest you post to the pyparsing pastebin at pyparsing.pastebin.com.

OK, here it is: :)
http://pyparsing.pastebin.com/d1390fd6b


> But I'll take a wild guess here.  I think "not" may be being interpreted by
> your grammar as a valid identifier.  Make sure your definition of operand
> includes something like:
>
> # be sure to define these as Keywords, and not Literals
> # if defined as Literals, an identifier like "notMyFathersOldsmobile"
> # would process the leading "not" as the "not" keyword
> true_ = Keyword("true")
> false_ = Keyword("false")
> and_ = Keyword("and")
> or_ = Keyword("or")
> not_ = Keyword("not")
>
> allKeywords = true_ | false_ | and_ | or_ | not_
>
> identifier = ~allKeywords + Word(alphas, alphanums+"_")
>
> operand = identifier | numeric_constant | quotedString

I'm using Suppress instead of Keyword because the identifiers are defined 
using a regular expression that allows any Unicode word character plus two 
special characters*, and with Keyword I'd have to pass in a string with all 
the valid characters (which I cannot do).

* One to separate namespaces (e.g., namespace:subnamespace, 
namespace/subnamespace, namespace.subnamespace) and another for spacing in 
identifiers (e.g., the_identifier, the-identifier, the+identifier).

> To double-check my theory, try using:
>
> operand.setDebug()
>
> And rerun your grammar to see if "not" is matching what you think it is
> matching.

I've checked it and the negation is not matched as an operand (because it's 
represented by the tilde (~), which is not a valid identifier).

I hope you can find the problem in the snippet above :)

Cheers!
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Indefinitely recursive grammar because of unaryoperation in operatorPrecedence()

[Paul M. <pt...@au...>]

Gustavo -

Once again, your attachments are not coming through the list server, so I
suggest you post to the pyparsing pastebin at pyparsing.pastebin.com.

But I'll take a wild guess here.  I think "not" may be being interpreted by
your grammar as a valid identifier.  Make sure your definition of operand
includes something like:

# be sure to define these as Keywords, and not Literals
# if defined as Literals, an identifier like "notMyFathersOldsmobile"
# would process the leading "not" as the "not" keyword
true_ = Keyword("true")
false_ = Keyword("false")
and_ = Keyword("and")
or_ = Keyword("or")
not_ = Keyword("not")

allKeywords = true_ | false_ | and_ | or_ | not_

identifier = ~allKeywords + Word(alphas, alphanums+"_")

operand = identifier | numeric_constant | quotedString


To double-check my theory, try using:

operand.setDebug()

And rerun your grammar to see if "not" is matching what you think it is
matching.

-- Paul

[Pyparsing] Indefinitely recursive grammar because of unary operation in operatorPrecedence()

[Gustavo N. <me...@gu...>]

Hello, everyone.

I'm using the operatorPrecedence() function to define a grammar for boolean 
expressions and this grammar becomes indefinitely recursive when I add an 
unary operation (the negation).

I'm using it like this:
    operation = operatorPrecedence(
        operand,
        [
            (relationals, 2, opAssoc.LEFT),
            (belongs_to, 2, opAssoc.LEFT),
            (is_subset, 2, opAssoc.LEFT),
            (not_, 1, opAssoc.RIGHT),
            (and_, 2, opAssoc.LEFT),
            (ex_or, 2, opAssoc.LEFT),
            (in_or, 2, opAssoc.LEFT),
        ]
    )

Because I want to support operations such as:
 * "pi == 3.1416" (relationals)
 * "3.1416 > pi" (relationals)
 * "user in allowed_people" (belongs_to)
 * "not user in allowed_people" (not_, belongs_to)
 * "not today_is_sunday" (not_)
 * "weather:today_will_rain() and weather:current_season == 'summer'"

This is, I really want relational and set-specific operations (belongs_to, 
is_subset) to have a higher precedence than the negation.

If I remove the not_ operator, the grammar is not indefinitely recursive 
anymore.

What am I doing wrong? I've extracted the relevant code from my project and 
I've attached it to this email, so you can reproduce it easily.

Thanks in advance!
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] Order problem in binary operation withoperationPrecedence()

[Paul M. <pt...@au...>]

> 
> I've defined a Pyparsing grammar for boolean expressions, based on
> operationPrecedence() and I have an strange problem.
> 
> <snip>
>
> What's going wrong? I've attached an excerpt from my library for you to
> reproduce it easily.
> 

Gustavo -

I'm afraid your attachments didn't make it through.  Could you paste your
code extract to http://pyparsing.pastebin.com and send us the link?

-- Paul

[Pyparsing] Order problem in binary operation with operationPrecedence()

[Gustavo N. <me...@gu...>]

Hello again, everybody.

I've defined a Pyparsing grammar for boolean expressions, based on 
operationPrecedence() and I have an strange problem.

This grammar has five types of operands: 3 literals (string, number, set) and 
2 non-literals (variable, function).

The following expressions work like a charm:
 * "pi == 3.1416"
 * "pi > 3.1416"
 * "pi < 3.1416"
 * "pi => 3.1416"
 * "pi <= 3.1416"

But if I rearrange the operands, I'd get an exception (like "Expected end of 
text (at char 7), (line:1, col:8)"):
 * "3.1416 == pi"
 * "3.1416 < pi"
 * "3.1416 > pi"
 * etc.

Every other combination of operands work, AFAIK, except when the first operand 
in a binary operation is a number:
 * "hello" in {"hi", "bye", "hello", "good-bye"}
 * "e < pi"
 * "pi > e"

What's going wrong? I've attached an excerpt from my library for you to 
reproduce it easily.

Thanks in advance!
-- 
Gustavo Narea <xri://=Gustavo>.
| Tech blog: =Gustavo/(+blog)/tech  ~  About me: =Gustavo/about |

Re: [Pyparsing] DSL using pyparsing

[Paul M. <pt...@au...>]

> -----Original Message-----
> From: Asim Malik [mailto:as...@ho...]
> Sent: Friday, June 12, 2009 6:31 AM
> To: pt...@au...; pyp...@li...
> Subject: RE: [Pyparsing] DSL using pyparsing
> 
> So is there a way for the import hook to somehow inject the import
> statements, when executing the code? Or does my generated code have to
> explicitly declare each import statement?
> 

Have your import hook inject the necessary import statements at the top of
the generated code:

    generated_code = """from Color import Color
                        %s""" % data

Here I put your Color class in a file named Color.py.

Now when I run the script testColor.py, which contains this code:

import Colors
import abc

print abc.c

I get:

from Color import Color


c = Color(255, 0, 255)

cols



Now the last step is for you to convert your assignment to generated_code to
use a pyparsing expression, and call transformString on the content of the
file (in your variable 'data').

-- Paul

Re: [Pyparsing] DSL using pyparsing

[Asim M. <as...@ho...>]

So is there a way for the import hook to somehow inject the import statements, when executing the code? Or does my generated code have to explicitly declare each import statement?

Let me try an break it down, suppose i have this code in a file 'Colors.py', the path of which is (xxx.dsl.Colors.py)  and for now the Color class resides here but it could be imported from some other external module:

import imputil

class Color(object):
    def __init__(self, r, g, b):
        self.r = r
        self.g = g
        self.b = b
    def __str__(self):
        return 'cols'

def convert_dsl(filepath, fileinfo, filename):
    srcfile = open(filepath, 'r')
    data = srcfile.read()
    srcfile.close()    

    generated_code = '%s' % data

    print generated_code

    COMPILE_MODE = 'exec'
    codeobj = compile(generated_code.rstrip(" \t"), 
                        filepath, 
                        COMPILE_MODE)
    
    return 0, codeobj, {}

importer = imputil.ImportManager()
importer.add_suffix('.col', convert_dsl)
importer.install()  --- Is this all that needs to be done or do i have to add some namespace or something?

Then i have my abc.col file:

c = Color(255, 0, 255)

In reality this would be a file which has the dsl that would be py-parsed, so obviously i don't want an import statement in this. If i did add 'from Colors import Color' at the top it works, but in reality i am expecting 'from xxx.dsl.Colors import Color'

Now if i run this, i get:
from xxx.dsl.Colors import *
import abc

print abc.c

I get the error:
    c = Color(255, 0, 255)
NameError: name 'Color' is not defined. I am not quite sure where the imports should be defined or resolved from.


> From: pt...@au...
> To: as...@ho...; pyp...@li...
> Subject: RE: [Pyparsing] DSL using pyparsing
> Date: Thu, 11 Jun 2009 21:11:56 -0500
> 
> > Thanks Paul, I think i have the idea now. I am using Imputil, so the
> > problem i have now is that i am using imputil, so when say i am trying to
> > compile a script, say:
> > 
> > A = Color(100, 10, 50)
> > 
> > I get a "NameError: name 'Color' is not defined". Any ideas?
> > 
> > 
> 
> Well, you *are* generating Python code, so if you had originally written
> that line of code you would get the same error.  The Color class needs to
> come from somewhere.  Either import it from another module, or define it
> somewhere in your source program (maybe in the module you import to include
> the imputil hook).
> 
> -- Paul
> 
> 

Re: [Pyparsing] DSL using pyparsing

[Paul M. <pt...@au...>]

> Thanks Paul, I think i have the idea now. I am using Imputil, so the
> problem i have now is that i am using imputil, so when say i am trying to
> compile a script, say:
> 
> A = Color(100, 10, 50)
> 
> I get a "NameError: name 'Color' is not defined". Any ideas?
> 
> 

Well, you *are* generating Python code, so if you had originally written
that line of code you would get the same error.  The Color class needs to
come from somewhere.  Either import it from another module, or define it
somewhere in your source program (maybe in the module you import to include
the imputil hook).

-- Paul

Re: [Pyparsing] DSL using pyparsing

[Asim M. <as...@ho...>]

Thanks Paul, I think i have the idea now. I am using Imputil, so the problem i have now is that i am using imputil, so when say i am trying to compile a script, say:
 
A = Color(100, 10, 50)

I get a "NameError: name 'Color' is not defined". Any ideas?


> From: pt...@au...
> To: as...@ho...; pyp...@li...
> Subject: RE: [Pyparsing] DSL using pyparsing
> Date: Wed, 10 Jun 2009 14:17:50 -0500
> 
> Asim -
> 
> I would work on this like they dig a tunnel - from both directions.  First I
> would write out what my DSL syntax would be.  Then I would work backwards
> and write out how this would be implemented in Python.  The purpose of the
> pyparsing DSL converter is to find your DSL syntax, and then create the
> related Python code.
> 
> Let's say your application was something about RGB colors, and A and B are
> going to be colors which, when added together, result in a third color whose
> RGB values are the respective sums of the R, G, and B values of A and B.
> Let's say this is your color syntax:
> 
> A = < 100 10 50 >
> B = < 0 0 0 >
> C = A+B
> 
> The purpose of the DSL converter is to convert the "<>" syntax into a
> Python-compatible API call (probably a constructor call to a Color class).
> The "C=A+B" line probably doesn't need DSL handling, your Python Color class
> can define the __add__() function to do the proper R, G, and B summing.
> 
> So all your DSL converter in this case would need to do would be to change:
> 
> A = < 100 10 50 >
> 
> To:
> 
> A = Color(100, 10, 50)
> 
> In pyparsing this is a very simple thing:
> 
> LT,GT = map(Suppress,"<>")
> Integer = Word(nums)
> colorDef = LT + integer*3 + RT
> colorDef.setParseAction(lambda tokens : "Color(%s, %s, %s)" % tokens)
> 
> Then use colorDef.transformString() to convert the source read from the
> imported file into valid Python, and then use that to compile into the
> executable module.
> 
> Remember, you are generating a Python script from your DSL script, so you
> don't have to worry about "carrying values forward from one line to the
> next" - when your generated code gets compiled to Python bytecode, it will
> do this for you.
> 
> Don't try to re-invent Python with your DSL.  You are *much* better off
> using your DSL to *augment* the Python syntax.
> 
> Thanks for reading the article!
> 
> -- Paul
> 
> 
> > -----Original Message-----
> > From: Asim Malik [mailto:as...@ho...]
> > Sent: Wednesday, June 10, 2009 1:58 PM
> > To: pyp...@li...
> > Subject: [Pyparsing] DSL using pyparsing
> > 
> > 
> > Ok i am looking at building a DSL using pyparsing and am relatively new to
> > python . Intially I am looking at something like this:
> > 
> > A = {build some list, this calls some underlying python api and creates
> > say a class A}
> > B = {build another list, this creates another python class B }
> > C= A+B
> > 
> > Should i parse this as a single script or parse each statement? If i parse
> > each statement how can i pass variables that were defined in the previous
> > statement? As the complexity of the doamin increase, it might be the case
> > that the construction of A depends on other statements. What i am looking
> > for is the best approach.
> > 
> > Thanks
> > 
> > 
> > 
> > --------------------------------------------------------------------------
> > ----
> > Crystal Reports - New Free Runtime and 30 Day Trial
> > Check out the new simplified licensing option that enables unlimited
> > royalty-free distribution of the report engine for externally facing
> > server and web deployment.
> > http://p.sf.net/sfu/businessobjects
> > _______________________________________________
> > Pyparsing-users mailing list
> > Pyp...@li...
> > https://lists.sourceforge.net/lists/listinfo/pyparsing-users
> 

Re: [Pyparsing] Combine problem

[Sebastian S. <ssc...@gm...>]

Thanks a lot for the quick replies!!

I didn't know about the , adjacent=False) argument for Combine.

['float3tan:TEXCOORD5;'] wasn't really what I expected but I was simply
confused by the exception - thanks for the explanations!

Nice work in the  "Under development" section too :)


All the best,

seb



On Thu, Jun 11, 2009 at 1:09 PM, spir <den...@fr...> wrote:

> Le Thu, 11 Jun 2009 12:00:32 +0900,
> Sebastian Schoellhammer <ssc...@gm...> s'exprima ainsi:
>
> > Hello there!
> >
> > This is my first post to this list - first of all thanks for this awesome
> > module, I could not live without it these days!
> >
> >
> > My goal is to parse the following structure:
> > s = """
> > struct VS_OUTPUT {
> >     float4 pos        : SV_Position;
> >     float2 UV    : TEXCOORD0;
> >     float3 vVec    : TEXCOORD1;
> >     float3 lVec    : TEXCOORD2;
> >     float3 nor     : TEXCOORD3;
> >     float4 col     : TEXCOORD4;
> >     float3 tan     : TEXCOORD5;
> >     float4 lpos     : TEXCOORD6;
> >     float depth     : TEXCOORD7;
> >     float2 dof     : TEXCOORD8;
> >     float2 VelocityUV      : TEXCOORD9;
> > };"""
> >
> > entryString = "float3 tan     : TEXCOORD5;"
> >
> > def readStruct(s):
> >     header = Literal("struct VS_OUTPUT")+"{"
> >
> >     regs = Combine("TEXCOORD"+Word(nums)) | "SV_Position"
> >     vType = Combine("float"+Optional(Word(nums)))
> >     entry = vType+Word(alphanums)+":"+regs+";"
> >     struct = header + OneOrMore(entry) + "};"
> >
> >     print entry.parseString(entryString)
> >
> > This is working as is - but when i change
> >
> > entry = vType+Word(alphanums)+":"+regs+";"  to entry =
> > Combine(vType+Word(alphanums)+":"+regs+";")
> >
> > to make this into a single string, I get a parse Exception - it's not a
> big
> > problem but I just don't understand why it doesn't work in this case :)
> > Somehow I can't see the difference to where I use Combine in the lines
> > above, where it seems just fine.
>
> You should always post error messages to get help people help you. In this
> case:
>  [.............]
>  File
> "/usr/lib/python2.5/site-packages/pyparsing-1.5.1-py2.5.egg/pyparsing.py",
> line 1657, in parseImpl
>    raise exc
> pyparsing.ParseException: Expected W:(abcd...) (at char 6), (line:1, col:7)
>
> In other words: pyparsing steps on the space, expecting the second part of
> the pattern.
>
> If I remember well, Combine() invalidates the default "separator-skipping"
> mode of pyparsing (someone confirms?). Which is rather sensible, for
> otherwise it would join together bit that are separated in the source, which
> is probably not what we want in most cases.
>
> If you wish this anyway, just join the result manually:
>        result = entry.parseString(entryString)
>        print "".join(result)
> or rather set a post-parse action on entry:
>        entry.setParseAction(lambda node: ''.join(node))
> ==>
> ['float3tan:TEXCOORD5;']
>
> Is this really what you expect?
>
> Denis
> ------
> la vita e estrany
>
>
> ------------------------------------------------------------------------------
> Crystal Reports - New Free Runtime and 30 Day Trial
> Check out the new simplified licensing option that enables unlimited
> royalty-free distribution of the report engine for externally facing
> server and web deployment.
> http://p.sf.net/sfu/businessobjects
> _______________________________________________
> Pyparsing-users mailing list
> Pyp...@li...
> https://lists.sourceforge.net/lists/listinfo/pyparsing-users
>



-- 
Sebastian Schoellhammer

Sr. Technical Artist
Square Enix LTD
www.square-enix.com

Re: [Pyparsing] Combine problem

[spir <den...@fr...>]

Le Thu, 11 Jun 2009 12:00:32 +0900,
Sebastian Schoellhammer <ssc...@gm...> s'exprima ainsi:

> Hello there!
> 
> This is my first post to this list - first of all thanks for this awesome
> module, I could not live without it these days!
> 
> 
> My goal is to parse the following structure:
> s = """
> struct VS_OUTPUT {
>     float4 pos        : SV_Position;
>     float2 UV    : TEXCOORD0;
>     float3 vVec    : TEXCOORD1;
>     float3 lVec    : TEXCOORD2;
>     float3 nor     : TEXCOORD3;
>     float4 col     : TEXCOORD4;
>     float3 tan     : TEXCOORD5;
>     float4 lpos     : TEXCOORD6;
>     float depth     : TEXCOORD7;
>     float2 dof     : TEXCOORD8;
>     float2 VelocityUV      : TEXCOORD9;
> };"""
> 
> entryString = "float3 tan     : TEXCOORD5;"
> 
> def readStruct(s):
>     header = Literal("struct VS_OUTPUT")+"{"
> 
>     regs = Combine("TEXCOORD"+Word(nums)) | "SV_Position"
>     vType = Combine("float"+Optional(Word(nums)))
>     entry = vType+Word(alphanums)+":"+regs+";"
>     struct = header + OneOrMore(entry) + "};"
> 
>     print entry.parseString(entryString)
> 
> This is working as is - but when i change
> 
> entry = vType+Word(alphanums)+":"+regs+";"  to entry =
> Combine(vType+Word(alphanums)+":"+regs+";")
> 
> to make this into a single string, I get a parse Exception - it's not a big
> problem but I just don't understand why it doesn't work in this case :)
> Somehow I can't see the difference to where I use Combine in the lines
> above, where it seems just fine.

You should always post error messages to get help people help you. In this case:
  [.............]
  File "/usr/lib/python2.5/site-packages/pyparsing-1.5.1-py2.5.egg/pyparsing.py", line 1657, in parseImpl
    raise exc
pyparsing.ParseException: Expected W:(abcd...) (at char 6), (line:1, col:7)

In other words: pyparsing steps on the space, expecting the second part of the pattern.

If I remember well, Combine() invalidates the default "separator-skipping" mode of pyparsing (someone confirms?). Which is rather sensible, for otherwise it would join together bit that are separated in the source, which is probably not what we want in most cases.

If you wish this anyway, just join the result manually:
	result = entry.parseString(entryString)
 	print "".join(result)
or rather set a post-parse action on entry:
	entry.setParseAction(lambda node: ''.join(node))
==>
['float3tan:TEXCOORD5;']

Is this really what you expect?

Denis
------
la vita e estrany