pyparsing-users

[Pyparsing] Memory issues with 2.0.6

[Will M. <wil...@gm...>]

Hi,

My app has a fairly complex grammar to parse expressions. Up to 2.0.5 it
was working well. But in 2.0.6 it gets stuck parsing and starts eating
memory until the process is killed by the OS.

I've pinned pyparsing to 2.0.5 for now. I haven't done any debugging yet,
but I was wondering what had changed since 2.0.5 that could trigger this
kind of behaviour? Any known bugs?

Thanks in advance,

Will McGugan

Re: [Pyparsing] Memory issues with 2.0.6

[Andrea C. <an...@cd...>]

> starts eating memory until the process is killed by the OS.

Me too!

I have been wondering why all of a sudden my Travis unit tests were
failing, with the processes being killed.

~

[I also have a case in which a grammar worked (defined as: "parses
string s") in Python 2.7, but doesn't (syntax error with the same
string s) in >=3.3, and the specific errors changed with the latest
update of PyParsing.

http://github.com/AndreaCensi/contracts

So I guess something substantial changed in the latest release.

]


On Thu, Nov 19, 2015 at 7:42 PM, Will McGugan <wil...@gm...> wrote:
> Hi,
>
> My app has a fairly complex grammar to parse expressions. Up to 2.0.5 it
> was working well. But in 2.0.6 it gets stuck parsing and starts eating
> memory until the process is killed by the OS.
>
> I've pinned pyparsing to 2.0.5 for now. I haven't done any debugging yet,
> but I was wondering what had changed since 2.0.5 that could trigger this
> kind of behaviour? Any known bugs?
>
> Thanks in advance,
>
> Will McGugan
> ------------------------------------------------------------------------------
> _______________________________________________
> Pyparsing-users mailing list
> Pyp...@li...
> https://lists.sourceforge.net/lists/listinfo/pyparsing-users



-- 
Andrea Censi | http://andrea.caltech.edu | "Not all those who wander are lost."
research scientist @ LIDS / Massachusetts Institute of Technology

Re: [Pyparsing] Memory issues with 2.0.6

[Paul M. <pt...@au...>]

There were two logic changes made in 2.0.6:

- a bug in Or (operator ^) was fixed which handles the case where the
longest match fails because of a parse action raising an exception, but an
alternative shorter match succeeds; previously this would erroneously fail
to match, but now successfully returns the shorter alternative match

- a bug in Each (operator &) was fixed that would erroneously return
multiple matches of Optional expressions

There was one additional change that introduced a bug that only affects
users with unicode in their expressions.

If your grammar has complex expressions (especially recursive expressions)
using ^ or & operators, these new bugfixes may be the problem.

Andrea, I'll try to take a look at the PyContracts code that you posted and
see if any glaring areas jump out. will, I hope these descriptions will give
you some clues where to start looking in your grammar.  I can also post some
before-after snippets that you can patch into your versions of pyparsing and
rerun your tests.

-- Paul


-----Original Message-----
From: Andrea Censi [mailto:an...@cd...] 
Sent: Thursday, November 19, 2015 7:03 PM
To: Will McGugan <wil...@gm...>
Cc: Pyp...@li...
Subject: Re: [Pyparsing] Memory issues with 2.0.6

> starts eating memory until the process is killed by the OS.

Me too!

I have been wondering why all of a sudden my Travis unit tests were failing,
with the processes being killed.
<snip>


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: [Pyparsing] Memory issues with 2.0.6

[Paul M. <pt...@au...>]

The problem is definitely in the "cosmetic-only" change to the returned
error message for MatchFirst and Or (which also manifests as a Unicode
error), and does not even require calling parseString, just streamline().
Thanks for the test case Will, I can repro the problem with it, but am
trying to distill it down to a smaller case to add to my unit tests, and to
work with in fixing the bug.

For now, impatient users can comment out line 2354 in pyparsing.py:

self.errmsg = "Expected " + str(self)


-- Paul


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

Re: [Pyparsing] Memory issues with 2.0.6

[Andrea C. <an...@cd...>]

On Fri, Nov 20, 2015 at 4:16 AM, Paul McGuire <pt...@au...> wrote:
>
> Andrea, I'll try to take a look at the PyContracts code that you posted and
> see if any glaring areas jump out. will, I hope these descriptions will give
> you some clues where to start looking in your grammar.  I can also post some
> before-after snippets that you can patch into your versions of pyparsing and
> rerun your tests.


Thanks Paul!

1) As for the PyContracts project, where I get syntax errors, this is
my travis project:

https://travis-ci.org/AndreaCensi/contracts

This currently shows the same code working for 2.7,3.2,3.3,pypy, but
failing for 3.4 and 3.5.

Currently the version that is installed by pip is  pyparsing-2.0.6-py2.py3.

I'm not an expert at Travis. I wish there was a way to run different
builds with multiple versions of pyparsing.

To look into the grammar, start here:
https://github.com/AndreaCensi/contracts/blob/master/src/contracts/syntax.py

Other parts are in other files. It is fairly complex - I was using
almost all the features of PyParsing.

In the past, what was failing in >=3.4 were tests related to the unary
operator "-". It didn't recognize things like ">=-1". Now the problem
is errors like this:

'array(=4|>=2,<=0)'
=>
pyparsing.ParseException: Expected {{FollowedBy:({{Forward:
{{FollowedBy:({{Forward: {{FollowedBy:({{... "*"} ...}) Group:({...
{{"*" ...}}...})} | ...} "-"} Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}) Group:({Forward:
{{FollowedBy:({{... "*"} ...}) Group:({... {{"*" ...}}...})} | ...}
{{"-" Forward: {{FollowedBy:({{... "*"} ...}) Group:({... {{"*"
...}}...})} | ...}}}...})} | Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}} "+"} Forward:
{{FollowedBy:({{Forward: {{FollowedBy:({{... "*"} ...}) Group:({...
{{"*" ...}}...})} | ...} "-"} Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}) Group:({Forward:
{{FollowedBy:({{... "*"} ...}) Group:({... {{"*" ...}}...})} | ...}
{{"-" Forward: {{FollowedBy:({{... "*"} ...}) Group:({... {{"*"
...}}...})} | ...}}}...})} | Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}}) Group:({Forward:
{{FollowedBy:({{Forward: {{FollowedBy:({{... "*"} ...}) Group:({...
{{"*" ...}}...})} | ...} "-"} Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}) Group:({Forward:
{{FollowedBy:({{... "*"} ...}) Group:({... {{"*" ...}}...})} | ...}
{{"-" Forward: {{FollowedBy:({{... "*"} ...}) Group:({... {{"*"
...}}...})} | ...}}}...})} | Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}} {{"+" Forward:
{{FollowedBy:({{Forward: {{FollowedBy:({{... "*"} ...}) Group:({...
{{"*" ...}}...})} | ...} "-"} Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}) Group:({Forward:
{{FollowedBy:({{... "*"} ...}) Group:({... {{"*" ...}}...})} | ...}
{{"-" Forward: {{FollowedBy:({{... "*"} ...}) Group:({... {{"*"
...}}...})} | ...}}}...})} | Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}}}...})} | Forward:
{{FollowedBy:({{Forward: {{FollowedBy:({{... "*"} ...}) Group:({...
{{"*" ...}}...})} | ...} "-"} Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}) Group:({Forward:
{{FollowedBy:({{... "*"} ...}) Group:({... {{"*" ...}}...})} | ...}
{{"-" Forward: {{FollowedBy:({{... "*"} ...}) Group:({... {{"*"
...}}...})} | ...}}}...})} | Forward: {{FollowedBy:({{... "*"} ...})
Group:({... {{"*" ...}}...})} | ...}}} (at char 10), (line:1, col:11)


2) As for the other project, the one with out of memory errors, I do
have unicode expressions in literals.
Could the changes in unicode expressions lead to out of memory errors?

~

Just read your last message about the cosmetic bug. When you push out
the fix, I will let you know if the problems above disappear.