pyparsing-users — User notes and help on the pyparsing module

[Pyparsing] problem installing pyparsing

[Isabelle C. <isa...@in...>]

Hi,

I have a problem installing pyparsing (v 1.4.11):

$ sudo python setup.py install
running install
error: invalid Python installation: unable to open 
/usr/lib/python2.5/config/Makefile (No such file or directory)

I do have a directory /usr/lib/python2.5, but I don't have directory 
config/ under it.

Do you know what's wrong ?

My configuration: Mandriva 2008.0 and rpm python is installed (version 
2.5.1)

Best regards,
Isabelle Cabrera

Re: [Pyparsing] Unparsed cruft

[Paul M. <pa...@al...>]

Yes, this is in the FAQ's.  Add a StringEnd() to the end of your grammar,
telling pyparsing that at the end of the grammar, you expect to be at the
end of the string.

Word(nums).parseString("123 abc") -> ['123']
(Word(nums) + StringEnd()).parseString("123 abc") -> raises ParseException

-- Paul
 

-----Original Message-----
From: pyp...@li...
[mailto:pyp...@li...] On Behalf Of Gre7g
Luterman
Sent: Saturday, March 08, 2008 11:22 AM
To: pyp...@li...
Subject: [Pyparsing] Unparsed cruft

My parser seems to parse everything correctly in a well-formed source file,
which is great, but it seems to stumble a bit if the user's input is
imperfect.

Here's an example link: 

http://pastie.textmate.org/163282

This example tries to parse an invalid expression. The input is "x + 5 y"
and it returns [['x', '+', 5]]. So it's able to handle everything that is
well-formed, but instead of throwing an exception when it hits the part it
can't parse, it just returns.

This would be real problematic for me. If I can't parse all of the user's
program, I need to know that instead of thinking that compilation was
successful.
Is there some way to determine how much of the string was actually parsed,
or to force an exception if the string cannot be parsed in its entirety?

Thanks,

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around
http://mail.yahoo.com 

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges.
Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Pyparsing-users mailing list
Pyp...@li...
https://lists.sourceforge.net/lists/listinfo/pyparsing-users

[Pyparsing] Unparsed cruft

[Gre7g L. <haf...@ya...>]

My parser seems to parse everything correctly in a
well-formed source file, which is great, but it seems
to stumble a bit if the user's input is imperfect.

Here's an example link: 

http://pastie.textmate.org/163282

This example tries to parse an invalid expression. The
input is "x + 5 y" and it returns [['x', '+', 5]]. So
it's able to handle everything that is well-formed,
but instead of throwing an exception when it hits the
part it can't parse, it just returns.

This would be real problematic for me. If I can't
parse all of the user's program, I need to know that
instead of thinking that compilation was successful.
Is there some way to determine how much of the string
was actually parsed, or to force an exception if the
string cannot be parsed in its entirety?

Thanks,

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pyparsing] Grouping when using asList()

[Ralph C. <ra...@in...>]

Hi Paul,

> Oooh, I'm *tempted*, so tempted.
> 
> What do you think of Joshua's proposal of passing through the **kw dict?
> It's a little more explicit without requiring the full ".setResultsName"
> syntax.  Side-by-side, the options are:
> 
> A) date = twoDigitInt("month*") + '/' + twoDigitInt("day")
> B) date = twoDigitInt("month", listAllMatches=True) + '/' +
> twoDigitInt("day")

I think that 'listAllMatches = True' is just noisy and interrupts the
flow.

> Mostly I'm concerned about accidental listAllMatches behavior when a
> results name is set from within the parsing of a Dict, which is a bit
> out of the developer's control - the results names are taken from the
> input text stream, and could reasonably contain any of these options.

OK.  What about a second optional argument that's not a long keyword?
Or guarantee that listAllMatches is it.  Any truth value should do.

    dd('month', 1) + '/' + dd('day')

Cheers,


Ralph.

Re: [Pyparsing] PEP-8 compliant naming, using synonyms

[Ralph C. <ra...@in...>]

Hi Paul,

> So I've considered adding PEP-8 compliant synonym definitions to
> pyparsing.  This would consist of a series of name bindings added to
> pyparsing that look like:
> 
> one_of = oneOf
> quoted_string = quotedString
> delimited_list = delimitedList
> rest_of_line = restOfLine

I think it's well worth doing this.  First impressions count, and seeing
non-PEP-8 stuff is a distraction.

The idea of issuing a deprecated warning in the future is a good one
too.

Cheers,


Ralph.

Re: [Pyparsing] Is a ternary operator possible?

[Gre7g L. <haf...@ya...>]

--- Paul McGuire <pa...@al...> wrote:

> Greg -
> 
> I'll give this some thought, get back to you in the
> next day or so.
> 
> In the meantime, look at operatorPrecedence and see
> what you think could be
> done to have it accept a ternary operator
> definition.

Well, you could do something like:

http://pastie.textmate.org/163086

Seems to work for me, but I haven't tested it
extensively.

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

[Pyparsing] Is a ternary operator possible?

[Gre7g L. <haf...@ya...>]

My parser is almost complete now. Everything seems to
be working great. Thanks again.

In the process, I had intentionally skipped over
trying to implement C's ternary operator (X ? Y : Z)
because operatorPrecedence can only handle one and two
operand commands.

Now, strictly speaking, I do not have to implement the
ternary operator, but if it won't take much effort, I
certainly would like to. Is easily do-able somehow?

I tried playing the same game with it as you showed me
for function calls -- namely defining a ternary
operation and then handling it in operatorPrecedence's
first parameter -- but this seems to recurse
infinitely (well, until maximum recursion depth is
reached).

Here's a simple sample program:

http://pastie.textmate.org/163034

Should I try something else or leave the ternary
operator out?

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pyparsing] A PyParsing hack

[Paul M. <pa...@al...>]

Greg - 

Bwa-ha-ha! Another minion in my evil domain!!! :)  I'm very happy that you
find pyparsing so useful AND enjoyable.

Your concept of passing a class instead of a function to setParseAction
looks pretty good.  Your point is dead-on - why bother returning just a
nested list of strings when you can use parse actions to construct and
return executable objects?  It is an excellent combination of both lexing
and parsing.  And extra credit for using parse actions as part of an
operatorPrecedence expression.

Please don't be disappointed, but yes, pyparsing *can* do this already.  In
fact, the code you posted/pasted, if you just change PP.NoStringBase to
object as the base class for your various parse action/classes, will run
otherwise unchanged with the unpatched pyparsing running on my system.  I
get this result:

[<Assignment 
	<Token 'X'> = <Math [<Token 'Y'>, 
				   '+', 
				   <Math [5, '*', <Token 'Z'>]
                           >]
                    >
>]

For another example of this technique, see:
http://pyparsing.wikispaces.com/space/showimage/simpleBool.py.  In addition
to being online at the pyparsing wiki, this script is included in the
examples directory of the source and doc distibutions.

In "Getting Started With Pyparsing"
(http://www.oreilly.com/catalog/9780596514235), the final chapter shows how
to convert query string expressions into objects that generate Python set
expressions, which can then be intersect'ed and union'ed to correspond to
the and's and or's of the original query string.  And I have a new magazine
article coming out soon that uses this method to compile an esoteric
language to executable objects.

Congratulations on arriving at this powerful concept on your own, it is most
definitely an advanced technique!

And keep proposing patches!  There are now many more brains than just my own
thinking about this module and how it could be improved, and I would be
foolish to limit new work to only the stuff I come up with myself.  Many of
the recent enhancements came as a result of discussions and patches
submitted over discussion lists and forums.  I'm glad that you are not
afraid to get your fingernails dirty, and get right into the pyparsing code
to see what it is doing.

-- Paul

(I'm sorry about unitTests - some of the files that I parse in my testing
are proprietary to some of my business customers, so I can't post them.
I'll try to update the SVN with what is publicly shareable, and
conditionalize the unitTests.py script to step over the private tests.)

[Pyparsing] A PyParsing hack

[Gre7g L. <haf...@ya...>]

First off, as a new user of PyParsing, I've gotta' say
that I am hooked. This thing rocks and is better than
sliced bread in a variety of ways.

Next, it does almost everything I could ever want...
but because it's open source, I'm able to make changes
to make it suit me. How cool is that? It's like every
day is my birthday, or something.

So, what follows is not really a question. I just
wanted to share what I'm doing and to make this
available to anyone else who might need to do
something similar. I guess if there is any element of
question to this e-mail, it would be, "Do you think
this is the cleanest way to add this functionality in
(assuming that it wasn't already here and I missed it
somehow)?" and if so, "How can I contribute this back
to the project?"

On to the show...

As I mentioned yesterday, I'm using PyParsing to parse
user code into something that my hardware can run. I
want to give the user a stripped-down version of C,
but due to extreme space limitations in my hardware, I
can't use any pre-existing virtualization code. So
let's say the user entered the function:

X=Y+5*Z

I would want to create instances that would represent
the various pieces of the instruction (one for each
variable, one for the multiplication, one for the
addition, and one for the assignment) and then call
some recursive member of the assignment instance to
generate the op-codes required to load/store and
calculate.

Teaching PyParsing to parse the assignment is pretty
trivial. With minimal effort, I can get it to churn
out the hierarchial list:

[['X', '=', ['Y', '+', [5, '*', 'Z']]]]

At this point, I *COULD* traverse back through the
output and convert the various pieces to instances,
but in essence, I would be doing the parsing process
twice. What I really would like is for PyParsing to
use its slick-as-snot setParseAction function to name
the class for me so I can just sit back and enjoy.

Here's a patch you can apply to the SVN head to make
setParseAction accept classes in addition to
functions:

http://pastie.textmate.org/162921

It only really does two things:

1. It accepts the class in lieu of a function, and
2. It provides a base class that the user can derive
from to prevent a reduce-to-text operation during the
parsing process.

With this code in place, the assignment is now
returned as (formatting added manually):

[
  <Assignment 
    <Token 'X'> =
      <Math
        [<Token 'Y'>, '+',
          <Math [5, '*', <Token 'Z'>]>
        ]
      >
  >
]

And finally, here's a really simple program to do the
above:

http://pastie.textmate.org/162922

My actual code handles a lot more kinds of code
structures and is still being debugged, but I didn't
want to bog everyone down in that, hence the tiny
snippet.

Finally, I tried to run the unitTests.py file to
verify that I hadn't broken anything in the process,
but I don't see all the required files in SVN.

Thanks again, 

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pyparsing] PEP-8 compliant naming, using synonyms

[Joshua J. K. <jk...@sk...>]

On Fri, 2008-03-07 at 08:34 -0600, Paul McGuire wrote:
> A while back, there was a suggestion to adopt PEP-8 compliant names in
> pyparsing.  For example, oneOf, delimitedList, and quotedString do not
> conform to the PEP-8 recommended naming style.  According to PEP-8, these
> should be one_of, delimited_list, and quoted_string.  For this discussion,
> I'll call these two naming styles "mixedCase" and "names_with_underscores".
> <SNIP>
> Any comments?

Funny you should bring this up.  When I first started reading the docs
for pyparsing, one of my first (albeit fleeting) thoughts was, "Hmm,
interesting, he doesn't follow PEP-8, the module must have been written
in the early days of Python before PEP-8 was well established."  I've
gone through a lot of naming conventions in my programming experience.
mostly_underscores when I did a lot of perl, MixedCase and
varHungarianNotation (shudder) when I did VB programming, and even mixed
case when I was starting in Python before I read PEP-8.

I'm not a fanatic of one style over another, really, and in fact kind of
like the look of MixedCase.  But I do like consistency, and if there is
a "blessed" way of doing things (as I would consider PEP-8 to be) I
would tend to lean in that direction.  In addition to the reason
regarding vision-imparied programmers, I would also add that MixedCase,
even if it is capitalizing every word, could still be harder for
non-native English speakers to read than words separated by underscores.

So, in short, I support the aliasing, and possibly the eventual
replacement, with PEP-8 compliant names.  Implementation idea: rename
all the functions to PEP-8 compliant names, and make wrappers using the
old names.  Then the wrappers would print out 'suchAndSuch deprecated,
use such_and_such instead.

Something else to thing about: what does PEP-8 say about keyword
arguments?  Like I said, I don't really prefer one over the other, but I
do like consistency.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Grouping when using asList()

[Joshua J. K. <jk...@sk...>]

On Fri, 2008-03-07 at 08:06 -0600, Paul McGuire wrote:
> Oooh, I'm *tempted*, so tempted.

Syntactical sugar is *always* tempting, I've found, even when it's
something that could come back to bite you later. :)

> Mostly I'm concerned about accidental listAllMatches behavior when a results
> name is set from within the parsing of a Dict, which is a bit out of the
> developer's control - the results names are taken from the input text
> stream, and could reasonably contain any of these options.

I would (personally) prefer the explicit listAllMatches=True simply
because you climb on to very shaky ground when you start introducing
special characters...they are kind of like "magic numbers."

And then there is the concept of "surprising behavior."  If I was
reading over someone's code (and was unfamiliar with pyparsing), and I
saw 'month*', I might be expecting it to do something like a wildcard
match, or some such.  The other alternatives seem a bit ambiguous, even
if [] does imply putting all the results in an array.

Actually, listAllMatches is a bit ambiguous too...keepAllMatches might
be a little bit clearer as far as that goes, but that is neither here
nor there.

So, my vote is to keep the explicitness, both for practical reasons and
readability/understanding-the-code reasons.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

[Pyparsing] PEP-8 compliant naming, using synonyms

[Paul M. <pt...@au...>]

A while back, there was a suggestion to adopt PEP-8 compliant names in
pyparsing.  For example, oneOf, delimitedList, and quotedString do not
conform to the PEP-8 recommended naming style.  According to PEP-8, these
should be one_of, delimited_list, and quoted_string.  For this discussion,
I'll call these two naming styles "mixedCase" and "names_with_underscores".

I've used names_with_underscores for many years, in Pascal, PL/1, and C.
When I learned Smalltalk, Java and C++, the general convention was shifting
(to my perception anyway) to mixedCase naming.  When I learned Tcl for my
job, names_with_underscores was again prevalent, and I eventually came to
equate mixedCase with "new and fun" and names_with_underscores with "archaic
and nasty."

So when I was developing pyparsing, I read PEP-8.  I was still fairly new to
Python, and had not fully navigated the waters of what's new and what's old.
Everything that's ever been done in Python is documented *somewhere*, and
very few places guide you that "module X is outmoded/deprecated" or "module
Z is the new module Y".  So I just assumed that PEP-8 was a product of its
own generation (as are we all), and reflected late-80's/early-90's thinking,
and that things had evolved some since the PEP's initial writing.  Wanting
to write code for the 21st century, I used mixedCase for pyparsing.

Since then, I've gotten a few isolated feedback comments that pyparsing is
nonstandard because it does not use names_with_underscores as directed in
PEP-8.  Of course, as any politician will tell you, the feedback you
actually get is usually representative of many more with the same opinion,
who for whatever reason choose not to express it aloud.  Unfortunately, much
of this discussion hinges on personal taste, so objective discussion is
usually contrived, or constructed by someone who is trying to rationalize
their existing preference (very similar to some of the "justifications" for
different bracing styles when trying to establish a C/C++ or Java coding
standard).  I did receive one actual, objective, practical distinction
between the two naming styles from one user who is vision impaired - on his
tactile- and audio-assist hardware, it is difficult to change case within a
contiguous word, but quite easy to insert an underscore, so pyparsing's use
of mixedCase really slows him down.

So I've considered adding PEP-8 compliant synonym definitions to pyparsing.
This would consist of a series of name bindings added to pyparsing that look
like:

one_of = oneOf
quoted_string = quotedString
delimited_list = delimitedList
rest_of_line = restOfLine

Existing names would not be deprecated unless there was some huge "hurrah!"
from the Python world.  But I know that many people are already using the
mixedCase style, and I strongly believe in not breaking peoples' existing
code.

Class names would not require synonyms, as ClassNameStyle as used in
pyparsing *is* PEP-8 compliant, so OneOrMore, ZeroOrMore, etc. would not
need to be changed or synonymified.

Any comments?

Thanks,
-- Paul

Re: [Pyparsing] Grouping when using asList()

[Paul M. <pt...@au...>]

Oooh, I'm *tempted*, so tempted.

What do you think of Joshua's proposal of passing through the **kw dict?
It's a little more explicit without requiring the full ".setResultsName"
syntax.  Side-by-side, the options are:

A) date = twoDigitInt("month*") + '/' + twoDigitInt("day")
B) date = twoDigitInt("month", listAllMatches=True) + '/' +
twoDigitInt("day")

I could go either way.  Just for the sake of alternatives, I tried some
other string-based indicators:

C) date = twoDigitInt("month!") + '/' + twoDigitInt("day")
D) date = twoDigitInt("month+") + '/' + twoDigitInt("day")
E) date = twoDigitInt("[month]") + '/' + twoDigitInt("day")
F) date = twoDigitInt("{month}") + '/' + twoDigitInt("day")

but I'm not compelled to use any of these over '*'.  

Mostly I'm concerned about accidental listAllMatches behavior when a results
name is set from within the parsing of a Dict, which is a bit out of the
developer's control - the results names are taken from the input text
stream, and could reasonably contain any of these options.

Any other comments from others on the list?

-- Paul



-----Original Message-----
From: pyp...@li...
[mailto:pyp...@li...] On Behalf Of Ralph
Corderoy
Sent: Friday, March 07, 2008 4:07 AM
To: pyp...@li...
Subject: Re: [Pyparsing] Grouping when using asList()


Hi Paul,

> date = twoDigitInt("month") + '/' + twoDigitInt("day") + '/' +
> twoDigitInt("year")
> 
> Try to look past the "it's a function call" to "this is a lot easier 
> to specify a results name than that old way!".  (However, if you need 
> to add the listAllMatches attribute, this still requires the explicit 
> call to setResultsName.)

How about signifying in the string that listAllMatches should be true, e.g.
'month*'?

Cheers,


Ralph.


-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges.
Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Pyparsing-users mailing list
Pyp...@li...
https://lists.sourceforge.net/lists/listinfo/pyparsing-users

Re: [Pyparsing] Grouping when using asList()

[Ralph C. <ra...@in...>]

Hi Paul,

> date = twoDigitInt("month") + '/' + twoDigitInt("day") + '/' +
> twoDigitInt("year")
> 
> Try to look past the "it's a function call" to "this is a lot easier
> to specify a results name than that old way!".  (However, if you need
> to add the listAllMatches attribute, this still requires the explicit
> call to setResultsName.)

How about signifying in the string that listAllMatches should be true,
e.g. 'month*'?

Cheers,


Ralph.

Re: [Pyparsing] Grouping when using asList()

[Joshua J. K. <jk...@sk...>]

On Thu, 2008-03-06 at 16:08 -0600, Paul McGuire wrote:
> Try to look past the "it's a function call" to "this is a lot easier to
> specify a results name than that old way!".  (However, if you need to add
> the listAllMatches attribute, this still requires the explicit call to
> setResultsName.)

Just had a thought: why not allow the listAllMatches=True to be added to
the calls to pyparsing elements?  Such as:

p.delimitedList(bracket_quoted('key_columns', listAllMatches=True))

In fact, here you go.  Starting on line 1170 of pyparsing.py (doc
strings deleted for brevity)

    def __call__(self, name, **kw):
        return self.setResultsName(name, **kw)

And just advertise pp_element() as a shortcut for
pp_element.setResultsName().

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Naming elements based on their predecessor

[Joshua J. K. <jk...@sk...>]

On Thu, 2008-03-06 at 19:19 -0600, Paul McGuire wrote:
> >>>>
> Noticed something interesting: on a parsing object, an unknown attribute
> (such as 'asdfasdfsadf') will return '' (an empty string).  Is this intended
> behavior?  It makes testing for attributes via hasattr impossible.  I can
> always test for an empty string, of course, but I'd rather use exceptions
> and catch an AttributeError than an if/then block to see if the string is
> non-emtpy.
> >>>>
> 
> To test for the existence of an attribute:
> - if 'alsfjlsjafs' in results:
> - if results.alkjasdlfjslf == '':
>   or if results.laksjdfljsd:  # evaluating to '' is a boolean false value
> - results["sldflsjf"] will raise an exception if no such attribute

Right, I figured that out (testing on the empty value anyway), but it
surprised me that an invalid attribute didn't raise an exception.

Thanks again for all your help.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Naming elements based on their predecessor

[Paul M. <pa...@al...>]

>>>>
Noticed something interesting: on a parsing object, an unknown attribute
(such as 'asdfasdfsadf') will return '' (an empty string).  Is this intended
behavior?  It makes testing for attributes via hasattr impossible.  I can
always test for an empty string, of course, but I'd rather use exceptions
and catch an AttributeError than an if/then block to see if the string is
non-emtpy.
>>>>

To test for the existence of an attribute:
- if 'alsfjlsjafs' in results:
- if results.alkjasdlfjslf == '':
  or if results.laksjdfljsd:  # evaluating to '' is a boolean false value
- results["sldflsjf"] will raise an exception if no such attribute
 
-- Paul

Re: [Pyparsing] Naming elements based on their predecessor

[Joshua J. K. <jk...@sk...>]

On Thu, 2008-03-06 at 18:23 -0600, Paul McGuire wrote:
> You are the perfect straight-man!

Uh, thanks? :)

> What you describe is exactly what the Dict class is for.  Dict is something
> kind of unwieldy, so there is also a dictOf(key,value) helper method.

Ah...spiffy.

OK, so here's what I came up with:

ext_prop = (p.Combine(p.Keyword('exec') +
                      p.Keyword('sp_addextendedproperty'),
                      joinString=" ", adjacent=False) +
            p.delimitedList(p.Dict(p.Group(p.Keyword('N').suppress() +
                        p.QuotedString(quoteChar="'") +
                        p.Literal(',').suppress() +
                        p.Keyword('N').suppress() +
                        p.QuotedString(quoteChar="'"))))
            )


And it seems to work just fine.  Noticed something interesting: on a
parsing object, an unknown attribute (such as 'asdfasdfsadf') will
return '' (an empty string).  Is this intended behavior?  It makes
testing for attributes via hasattr impossible.  I can always test for an
empty string, of course, but I'd rather use exceptions and catch an
AttributeError than an if/then block to see if the string is non-emtpy.
But that's just me.

Anyway, thanks for the pointers.  You've been a *huge* help.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Stuck on recursive parsing

[Gre7g L. <haf...@ya...>]

Holy cow!  That works incredibly well.

Many, many thanks.

Gre7g

--- Paul McGuire <pa...@al...> wrote:

> Greg -
> 
> Welcome to pyparsing, and a nice first attempt for
> someone new!

<snipped>

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com 

Re: [Pyparsing] Naming elements based on their predecessor

[Paul M. <pa...@al...>]

Joshua -

You are the perfect straight-man!

What you describe is exactly what the Dict class is for.  Dict is something
kind of unwieldy, so there is also a dictOf(key,value) helper method.  

I'm in the middle of some work at the moment, but check out the dict
examples on the pyparsing wiki, or in the examples directory of your
pyparsing distribution, or google for "pyparsing dict" to find any previous
comments I've posted on it.

-- Paul

 

-----Original Message-----
From: pyp...@li...
[mailto:pyp...@li...] On Behalf Of Joshua
J. Kugler
Sent: Thursday, March 06, 2008 6:18 PM
To: PyParsing User List
Subject: [Pyparsing] Naming elements based on their predecessor

Now that I have your attention via that confusing subject line, here is what
I'm trying to do.  I read through the manual again...if what I'm describing
is in there, it didn't jump out at me.

In the MS SQL table dumps I'm parsing, there are lines such as these:

exec sp_addextendedproperty N'MS_Description', N'The user and password
table', N'user', N'dbo', N'table', N'auth_login'
GO
exec sp_addextendedproperty N'MS_Description', N'User name', N'user',
N'dbo', N'table', N'auth_login', N'column', N'login'
GO

I guess you could say they are kind of a really messed up implementation of
Python's keyword arguments.  After the procedure call, you have lines of the
format:

N'Key' N'Value'...

So, I have a Pyparsing expression that parses these nicely:

ext_prop = (p.Combine(p.Keyword('exec') +
                      p.Keyword('sp_addextendedproperty'),
                      joinString=" ", adjacent=False) +
            p.delimitedList(p.Keyword('N').suppress() +
                            p.QuotedString(quoteChar="'"))
            )

Combine that with:

table_dump = (create_table.setResultsName('table_def') +
              p.Keyword('GO').suppress() +
              p.ZeroOrMore(ext_prop.setResultsName('ext_prop',
listAllMatches=True)
                           + p.Keyword('GO').suppress())
             )

and
create_data = table_dump.parseString(sql)

for ep in create_data.ext_prop:
    print ep

Gives me:

['exec sp_addextendedproperty', 'MS_Description', 'The user and password
table', 'user', 'dbo', 'table', 'auth_login'] ['exec
sp_addextendedproperty', 'MS_Description', 'User name', 'user', 'dbo',
'table', 'auth_login', 'column', 'login']

Nice.  Now, I'd like to extend that in this way: Starting with the first
"key" (in this case MS_Description) I'd like the follow value to have the
name of that key, as if I had called setResultsName for it.  Said another
(hopefully clearer) way:  When I do that above "for ep..." loop, I'd like to
be able to:

print ep.MS_Description #and get 'User name'
print ep.table #and get auth_login
print ep.column #and get login

I'd check for ep.column first via hasattr, since it might not be there if
it's parsing a call to set exteneded properties for a table.

Is this at all doable without some major pyparsing hackery?  Granted, I can
just access by position in list, but I like the attribute accessors, as they
look cleaner and are somewhat self documenting.  Of course, I can always
spell out the full procedure call, and manually name each value, but the
less (and simpler) the code the better.

j

--
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer) S&K Aerospace Alaska

-------------------------------------------------------------------------
This SF.net email is sponsored by: Microsoft Defy all challenges.
Microsoft(R) Visual Studio 2008.
http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
_______________________________________________
Pyparsing-users mailing list
Pyp...@li...
https://lists.sourceforge.net/lists/listinfo/pyparsing-users

[Pyparsing] Naming elements based on their predecessor

[Joshua J. K. <jk...@sk...>]

Now that I have your attention via that confusing subject line, here is
what I'm trying to do.  I read through the manual again...if what I'm
describing is in there, it didn't jump out at me.

In the MS SQL table dumps I'm parsing, there are lines such as these:

exec sp_addextendedproperty N'MS_Description', N'The user and password table', N'user', N'dbo', N'table', N'auth_login'
GO
exec sp_addextendedproperty N'MS_Description', N'User name', N'user', N'dbo', N'table', N'auth_login', N'column', N'login'
GO

I guess you could say they are kind of a really messed up implementation
of Python's keyword arguments.  After the procedure call, you have lines
of the format:

N'Key' N'Value'...

So, I have a Pyparsing expression that parses these nicely:

ext_prop = (p.Combine(p.Keyword('exec') +
                      p.Keyword('sp_addextendedproperty'),
                      joinString=" ", adjacent=False) +
            p.delimitedList(p.Keyword('N').suppress() +
                            p.QuotedString(quoteChar="'"))
            )

Combine that with:

table_dump = (create_table.setResultsName('table_def') +
              p.Keyword('GO').suppress() +
              p.ZeroOrMore(ext_prop.setResultsName('ext_prop', listAllMatches=True)
                           + p.Keyword('GO').suppress())
             )

and
create_data = table_dump.parseString(sql)

for ep in create_data.ext_prop:
    print ep

Gives me:

['exec sp_addextendedproperty', 'MS_Description', 'The user and password table', 'user', 'dbo', 'table', 'auth_login']
['exec sp_addextendedproperty', 'MS_Description', 'User name', 'user', 'dbo', 'table', 'auth_login', 'column', 'login']

Nice.  Now, I'd like to extend that in this way: Starting with the first
"key" (in this case MS_Description) I'd like the follow value to have
the name of that key, as if I had called setResultsName for it.  Said
another (hopefully clearer) way:  When I do that above "for ep..." loop,
I'd like to be able to:

print ep.MS_Description #and get 'User name'
print ep.table #and get auth_login
print ep.column #and get login

I'd check for ep.column first via hasattr, since it might not be there
if it's parsing a call to set exteneded properties for a table.

Is this at all doable without some major pyparsing hackery?  Granted, I
can just access by position in list, but I like the attribute accessors,
as they look cleaner and are somewhat self documenting.  Of course, I
can always spell out the full procedure call, and manually name each
value, but the less (and simpler) the code the better.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Stuck on recursive parsing

[Paul M. <pa...@al...>]

Greg -

Welcome to pyparsing, and a nice first attempt for someone new!

Of course, the solution to this is to "trick" pyparsing into accepting a
function call.  First of all, I would suggest that your function call should
really be something like this:

Function = Token + "(" + PP.delimitedList(Expr2) + ")"

in order to handle functions that have multiple arguments.  However, if you
are just trying to support basic math functions like abs, sqrt, sin, etc.,
then what you have is fine.

Ok.  The next step is to expand your concept a little bit.  In your grammar,
a function call is really at the same expression level as an integer or a
token.  So let's insert a function as one of the possible terms that could
be evaluated within your expression:

Expr2 = PP.operatorPrecedence(Integer | Function | Token,
    [(PP.oneOf("not - +"), 1, PP.opAssoc.RIGHT),
     (PP.oneOf("* / %"), 2, PP.opAssoc.LEFT),
     ... etc.

(I've placed Function ahead of Token in the list, so that "abs(-6)" tries to
evaluate as a function first, and a token second.)

Ah, but Function was defined using Expr2, we can't use it in the definition
of Expr2 also.

Unless... you guessed it - we make Expr2 a Forward!  Like this:

Expr2 = PP.Forward()
Function = Token + "(" + PP.delimitedList(Expr2) + ")"

# Basic operations
# change '=' to '<<' here
Expr2 << PP.operatorPrecedence(Integer | Function | Token,
    [(PP.oneOf("not - +"), 1, PP.opAssoc.RIGHT),
     (PP.oneOf("* / %"), 2, PP.opAssoc.LEFT),
     ... etc.

At this point, we are getting *very* recursive.  You can try to evaluate
your test string, but it might take a *long* time to run.  I am working on
this issue with another pyparsing user, but at the moment, it is a
performance issue of operatorPrecedence.  To workaround this problem, we can
use pyparsing's packrat parsing support.  Immediately after your "import
pyparsing..." statement, add this line:

PP.ParserElement.enablePackrat()

This reduces a lot of the repetitive parsing calls, and can improve the
performance of a heavily recursive grammar by 1000X or more.

With these changes, I now get:

[[2, '-', [5, '*', 'abs', '(', ['-', 6], ')'], '+', 3]]

A couple parting thoughts:

It would be good to retain the association of 'abs' and its argument list,
instead of just having that flat list of parsed tokens.  Pyparsing's Group
class is for just this purpose.  Change the definition of Function to Group
the entire expression, and add another Group for the arguments (to make them
easy to pick out from within the function's token list), like this:

Function = PP.Group(Token + 
        PP.Group("(" + PP.delimitedList(Expr2) + ")"))

and your results are now better structured for easier processing:

[[2, '-', [5, '*', ['abs', ['(', ['-', 6], ')']]], '+', 3]]

Or with some better indentation:

[
  [2, '-', 
    [5, '*', 
      ['abs', 
        ['(', ['-', 6], ')']
      ]
    ], '+', 3
  ]
]

Lastly, here is an alternative way to represent your potentially
dot-delimited identifier token:

Ident = PP.Word(PP.alphas, PP.alphanums + "_")
Token = PP.Combine(Ident + PP.ZeroOrMore('.' + Ident))

This is just a style point, and what you have is working fine, but this
change avoids duplicating the 'PP.Word(PP.alphas, PP.alphanums + "_")' bit;
and adds the pyparsing Combine expression, which will collapse the separate
tokens of a reference like "math.abs" from ['math', '.', 'abs'] to just
'math.abs'.

You could also have done the repetition using the pyparsing delimitedList
helper, as:

Token = PP.delimitedList(Ident,'.',combine=True)

Again, this is much more personal style, and is completely dependent on your
own taste in using pyparsing.

Cheers!
-- Paul

Re: [Pyparsing] Grouping when using asList()

[Joshua J. K. <jk...@sk...>]

On Thu, 2008-03-06 at 16:08 -0600, Paul McGuire wrote:
> Try to look past the "it's a function call" to "this is a lot easier to
> specify a results name than that old way!".  (However, if you need to add
> the listAllMatches attribute, this still requires the explicit call to
> setResultsName.)
> 
> I hope that's not too confusing.

Makes perfect sense. Thanks for the explanation.

j

-- 
Joshua Kugler
VOC/SigNet Provider (aka Web App Programmer)
S&K Aerospace Alaska

Re: [Pyparsing] Grouping when using asList()

[Paul M. <pa...@al...>]

Joshua -

You already had most of the pieces there, just a couple of gaps around a
little-used feature.

>>>One question: ("defs") is a call to what Groups() returns, but I'm not
following. Can you point me to something in the docs that explains what is
done when you call a pyparsing expression?<<<

Let's say you have this form for a date string:

twoDigitInt = Word(nums,exact=2)
date = twoDigitInt + '/' + twoDigitInt + '/' + twoDigitInt

The old way to assign results names was (I'm using US date format mm/dd/yy
convention):

date = twoDigitInt.setResultsName("month") + '/' +
twoDigitInt.setResultsName("day") + '/' + twoDigitInt.setResultsName("year")

I *really* wanted to encourage people to use results names, so I made
ParserElements "callable", and having the __call__ function call
setResultsName, thus supporting the following shortened syntax:

date = twoDigitInt("month") + '/' + twoDigitInt("day") + '/' +
twoDigitInt("year")

Try to look past the "it's a function call" to "this is a lot easier to
specify a results name than that old way!".  (However, if you need to add
the listAllMatches attribute, this still requires the explicit call to
setResultsName.)

I hope that's not too confusing.

-- Paul

[Pyparsing] Stuck on recursive parsing

[Gre7g L. <haf...@ya...>]

Hey guys, I'm brand new to pyparsing and I'm already
stuck.

My boss wants me to make our product "scriptable", so
I'm trying to write some Python code to compile an
expression entered by the user into something the
hardware can execute. Basic op-codes such as +, -, *,
and / were easy enough to compile with
operatorPrecedence, but I got in trouble when I tried
to make my code handle pre-defined function calls.

I wrote this using Forward and <<, but pyparsing isn't
going for it. Here's a minimal fragment of code
illustrating what I was trying to do:

http://pastie.textmate.org/162526

As you can see, it's trying to compile the expression
"2-5*abs(-6)+3". I was hoping it would return
something like:

[[2, '-', [5, '*', ['abs' + '(' + ['-' + 6] + ')']]
    '+' 3]]

but instead I got:

[[2, '-', [5, '*', 'abs']]]

It looks kind of like operatorPrecedence interpretted
'abs' as a literal and then got confused when it hit
the parenthesis, and gave up.

Can anyone suggest a way to work around this?

Many thanks and my apologies if this has been
discussed at length. If so, I'd love a link to a
pertinent post.

Gre7g

__________________________________________________
Do You Yahoo!?
Tired of spam?  Yahoo! Mail has the best spam protection around 
http://mail.yahoo.com