Re: [Pyparsing] Memory issues with 2.0.6

[Martijn V. <m.v...@lu...>]

Dear Paul,

Thanks, the memory issue seems to be resolved.

I do have another problem with 2.0.6 and up which I think is due to the
same change.

On parsing something that's not accepted by the grammar, I get a parse
exception, for example:

   ParseException: Expected "IVS" (at char 7), (line:1, col:8)

With the latest SVN I get the same thing, but the exception message
contains a huge expected string (more than 600,000 characters), for
example:

  ParseException: Expected {{{{{{Suppress:({["GI"] ^ ["GI:"] ^ ["gi"] ^
["gi:"]}) W:(0123...)} ^ {~{"LRG_"} Combine:({W:(abcd...) W:(0123...)})
[{Suppress:(".") W:(0123...)}]} ^ Combine:({"UD_" W:(abcd...) {{"_"
W:(0123...)}}...})} [{Suppress:("(") Group:({W:(abcd...)
[{{Suppress:("_v") W:(0123...)} ^ {Suppress:("_i") W:(0123...)}}]})
Suppress:(")")}]} ^ {Combine:({"LRG_" W:(0123...)}) [{{Suppress:("t")
W:(0123...)} ^ {Suppress:("p") W:(0123...)}}]}} Suppress:(":")
[{W:(cgmn...) Suppress:(".")}] {Empty {Gro ...

etcetera, you get the idea.

I'd say that this is undesirable.

This is with the same HGVS grammar I linked earlier. Just try parsing
the empty string for example.

https://github.com/mutalyzer/mutalyzer/blob/master/mutalyzer/grammar.py

best,
Martijn



On Wed, 2015-11-25 at 14:00 -0600, Paul McGuire wrote:
> For those who reported having memory and Unicode issues, if possible, please
> download the latest committed version of pyparsing from the SourceForge SVN
> repo, and see if this resolves your issues.  
> 
> For the memory problem, it is probably not necessary to actually parse any
> text, simply invoke the streamline() method on your top-level grammar
> instance:
> 
>     parser.streamline()
> 
> For the Unicode problems, you should stop seeing the UnicodeEncodeError
> exceptions when creating your parser.
> 
> Thanks for the feedback, everyone - if these changes work out, I'll follow
> up with an actual 2.0.7 release in the next day or so.
> 
> -- Paul
> 
> 
> 
> -----Original Message-----
> From: Martijn Vermaat [mailto:m.v...@lu...] 
> Sent: Tuesday, November 24, 2015 10:32 AM
> To: pyp...@li...
> Subject: Re: [Pyparsing] Memory issues with 2.0.6
> 
> Dear Paul and others,
> 
> (Sorry for replying out of thread, I only just subscribed.)
> 
> I can report the same problem with Pyparsing 2.0.6 on this grammar:
> 
> https://github.com/mutalyzer/mutalyzer/blob/master/mutalyzer/grammar.py
> 
> Memory usage continues to increase.
> 
> best,
> Martijn
> 
> 
> > The problem is definitely in the "cosmetic-only" change to the 
> > returned error message for MatchFirst and Or (which also manifests as 
> > a Unicode error), and does not even require calling parseString, just 
> > streamline().  Thanks for the test case Will, I can repro the problem 
> > with it, but am trying to distill it down to a smaller case to add to 
> > my unit tests, and to work with in fixing the bug.
> >
> > For now, impatient users can comment out line 2354 in pyparsing.py:
> >
> > self.errmsg = "Expected " + str(self)
> 
> 
> 
> ----------------------------------------------------------------------------
> --
> Go from Idea to Many App Stores Faster with Intel(R) XDK Give your users
> amazing mobile app experiences with Intel(R) XDK.
> Use one codebase in this all-in-one HTML5 development environment.
> Design, debug & build mobile apps & 2D/3D high-impact games for multiple
> OSs.
> http://pubads.g.doubleclick.net/gampad/clk?id=254741551&iu=/4140
> _______________________________________________
> Pyparsing-users mailing list
> Pyp...@li...
> https://lists.sourceforge.net/lists/listinfo/pyparsing-users
> 
> 
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>