#3 Use of Generators is not thread safe


An instance of generator.Generator is not safe to use in multiple threads simultaneously, because its buildParser method mutates its parserList state. This in itself is not a problem, but simpleparsegrammar.Parser.buildTagger and objectgenerator.LibraryElement.toParser re-use the same generators even among separate parser.Parser instances.

I'm no expert in simpleparse, but my testing shows that this can be remedied with two small changes: first, for simpleparsegrammar.Parser.buildTagger, copy the SPGenerator object instead of using it directly:

def buildTagger( self, name=None, processor = None ):
"""Build the tag-table for parsing the EBNF for this parser"""
gen = generator.Generator()
gen.names = SPGenerator.names[:]
gen.rootObjects = SPGenerator.rootObjects[:]
gen.methodSource = SPGenerator.methodSource
gen.definitionSources = SPGenerator.definitionSources[:]

return gen.buildParser( name, processor )

Second, for objectgenerator.LibraryElement.toParser, make a new Parser instance for each LibraryElement() instance created in common/*.py, instead of re-using a common "_p" variable. For example, in numbers.py:

for name in ["int","hex", "int_unsigned", "number", "float", "binary_number", "float_floatexp", "imaginary_number", "number_full"]:
c[ name ] = objectgenerator.LibraryElement(
generator = Parser( declaration )._generator,
production = name,

The project is a bit complicated for me to know if this preserves semantics, but it works in my tests.


  • Robert Brewer

    Robert Brewer - 2012-06-01

    Hmm. Perhaps it doesn't work so well in my tests. A better approach might be to create a Builder object instead of mutating the state of the Generator itself.


Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks