[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

[Pyparsing] Parsing large files using SkipTo is slow.

[<Maximilian.Argus@dl...>]

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max

Re: [Pyparsing] Parsing large files using SkipTo is slow.

[Ralph Corderoy <ralph@in...>]

Hi Max,

Your email has reached the list, several times, please stop sending it.
:-)

Cheers, Ralph.

Re: [Pyparsing] Parsing large files using SkipTo is slow.

[Paul McGuire <ptmcg@au...>]

First off, you have some mixed spaces/tabs in your file, you should clean
that up.

By using '^' operators inside a Forward recursive expression, your parser is
doing a lot of extra work.

I got some better response by changing the commented lines below:

#~ value = Word(alphas) + ( ( sls + SkipTo(']',include=False)+srs ) ^
Word(alphas +"_") )
value = Word(alphas) + ( ( sls + SkipTo(']',include=False)+srs ) |
Word(alphas +"_") )
value.setParseAction(p_val)

named_block = Forward()
#~ block_content = OneOrMore( value ^ named_block )
block_content = OneOrMore( named_block | value )
named_block << OneOrMore(Word(alphas)) + slc + block_content + src
named_block.setParseAction( p_nb )
named_block.ignore(pythonStyleComment)
#~ named_block.enablePackrat()


There were 3.2 million calls to "hash", most likely due to the packrat
parsing. By changing Or's to MatchFirst's, the packratting was not
necessary, and was in fact costing extra performance.

-- Paul


-----Original Message-----
From: Maximilian.Argus@... [mailto:Maximilian.Argus@...] 
Sent: Monday, September 12, 2011 7:10 AM
To: pyparsing-users@...
Subject: [Pyparsing] Parsing large files using SkipTo is slow.

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max


----------------------------------------------------------------------------
--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
Pyparsing-users mailing list
Pyparsing-users@...
https://lists.sourceforge.net/lists/listinfo/pyparsing-users

Re: [Pyparsing] Parsing large files using SkipTo is slow.

[Paul McGuire <ptmcg@au...>]

Max -

The FTP link you posted does not respond for me.  Can you try posting code
and any ASCII files at pastebin.com?

-- Paul


-----Original Message-----
From: Maximilian.Argus@... [mailto:Maximilian.Argus@...] 
Sent: Monday, September 12, 2011 6:06 AM
To: pyparsing-users@...
Subject: [Pyparsing] Parsing large files using SkipTo is slow.

Hello,

 I am trying to parse a simple CAD file using pyparsing, where the
 grammar is quite simple and most of the lines of the file are supposed
 to be skipped. However this is still slower than I would expect it to
 be.

 The parser is in the parse_iv.py file,  the CAD file is the
 screwdriver_2.iv and the CProfile result is th err file.

 It seems like parseString is called for each line. I don't understand
 the internal workings of pyparsing but would it be possible to use
 something like file.index( skiptostr ) to speed things up?

ftp://ftp.robotic.dlr.de/outgoing/Argus/


 BR, Max


----------------------------------------------------------------------------
--
Doing More with Less: The Next Generation Virtual Desktop 
What are the key obstacles that have prevented many mid-market businesses
from deploying virtual desktops?   How do next-generation virtual desktops
provide companies an easier-to-deploy, easier-to-manage and more affordable
virtual desktop model.http://www.accelacomm.com/jaw/sfnl/114/51426474/
_______________________________________________
Pyparsing-users mailing list
Pyparsing-users@...
https://lists.sourceforge.net/lists/listinfo/pyparsing-users

Re: [Pyparsing] Parsing large files using SkipTo is slow.

[Ralph Corderoy <ralph@in...>]

Hi Paul,

> > ftp://ftp.robotic.dlr.de/outgoing/Argus/
>
> The FTP link you posted does not respond for me.  Can you try posting
> code and any ASCII files at pastebin.com?

Try ftp://ftp.dlr.de/robotic/outgoing/Argus/

Cheers, Ralph.