Re: [Pyparsing] Pyparsing, multiple threads & parseString

[Diez B. R. <de...@we...>]

Hi,


>  I do have small application which is using pyparsing from multiple
> threads. The pyparsing is singleton and also the actual parseString is
> inside Lock()s, so it should be thread safe. (Below cuts from the script)
> 
>  The problem is that after the parseBlock has returned the ParseResults
> for me, and I go through the whole list, I can not get free the the full
> ParseResult dictionaries and if my parseBlock is called quite many times
> with different scripts, I end in the situation where I do have huge amount
> of dictionaries (len(objgraph.by_type('dict'))) and loosing memory bit by
> bit. I have tried deleting the entries with del, but haven't fully figured
> out the correct way of cleaning the ParseResults. How could I do the
> deleting for returned ParseResults ?
> 
>  I have tested using both scanString and parseString for my case, but I
> think parseString would be more suitable. And both raises the memory usage.
> 
>  Thank you very much for any tips and huge thanks for pyparsing,

Can't comment on MT-fitness for pyparsing, but one thing I know for sure: Python multiprocessing module is a *blast*, and it will give you proper scaling and is easy to use.

So maybe you can shell out the parsing to a multiprocessing-worker, reaping the benefits of real parallelization + processes which can be destructed & re-created to deal with any memory issues whatsoever?

Diez