Re: [Pyparsing] Pyparsing, multiple threads & parseString

[Mika S. <mik...@gm...>]

Hi,

 Thanks for advice ! I certainly will be using multiprocessing module.

  To clarify a bit more about memory consumption problem. I am using latest
2.0.0 pyparsing and python 3.3.1. If I try using my class e.g. like below
(with scanString), I will get unwanted effect.

psp = parser.parseBlock(script)
for i in psp:
  print(i)

If I won't go through the generator/psp, I won't get raising memory effect.
I have tried removing my custom token class, but still no luck. Thanks for
advices

  -Mika




On Fri, Apr 12, 2013 at 12:01 PM, Diez B. Roggisch <de...@we...> wrote:

> Hi,
>
>
> >  I do have small application which is using pyparsing from multiple
> > threads. The pyparsing is singleton and also the actual parseString is
> > inside Lock()s, so it should be thread safe. (Below cuts from the script)
> >
> >  The problem is that after the parseBlock has returned the ParseResults
> > for me, and I go through the whole list, I can not get free the the full
> > ParseResult dictionaries and if my parseBlock is called quite many times
> > with different scripts, I end in the situation where I do have huge
> amount
> > of dictionaries (len(objgraph.by_type('dict'))) and loosing memory bit by
> > bit. I have tried deleting the entries with del, but haven't fully
> figured
> > out the correct way of cleaning the ParseResults. How could I do the
> > deleting for returned ParseResults ?
> >
> >  I have tested using both scanString and parseString for my case, but I
> > think parseString would be more suitable. And both raises the memory
> usage.
> >
> >  Thank you very much for any tips and huge thanks for pyparsing,
>
> Can't comment on MT-fitness for pyparsing, but one thing I know for sure:
> Python multiprocessing module is a *blast*, and it will give you proper
> scaling and is easy to use.
>
> So maybe you can shell out the parsing to a multiprocessing-worker,
> reaping the benefits of real parallelization + processes which can be
> destructed & re-created to deal with any memory issues whatsoever?
>
> Diez
>
>