Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

MaybeEncodingError with pyparsing.asDict when using multiprocessing.pool map()

Dave Rigby
2014-02-10
2014-02-10
  • Dave Rigby
    Dave Rigby
    2014-02-10

    I'm trying to use multiprocessing.pool to speed up some parsing of a file parsed using pyparsing, however I get a multiprocessing.pool.MaybeEncodingError exception whenever I try this.

    I've narrowed it down to something to do with returning a dictionary (ParseResults.asDict()), using asList() the error doesn't occur; but the input I'm actually parsing is pretty complex so ideally I'd like to use asDict.

    Simplified testcase:

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    #!/usr/bin/env python2.7
    from pyparsing import *
    import multiprocessing
    
    value = ( Word(alphas) )
    key_val = Group( value + ":" + value )
    lang = Dict( Suppress('[') + delimitedList( key_val ) + Suppress(']') )
    
    def parse_dict(s):
        return lang.parseString(s).asDict()
    
    def parse_list(s):
        return lang.parseString(s).asList()
    
    data = ['[ foo : bar ]']
    pool = multiprocessing.Pool()
    
    # This works (list)
    pool.map(parse_list, data)
    
    # This fails (dict)
    pool.map(parse_dict, data)
    

    Fails with:

    Traceback (most recent call last):
      File "lib/python/nutshell/multi_parse.py", line 19, in <module>
        pool.map(parse, data)
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 250, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 554, in get
        raise self._value
    multiprocessing.pool.MaybeEncodingError: Error sending result: '[{'foo': ([':', 'bar'], {})}]'. Reason: 'TypeError("'str' object is not callable",)'
    

    Anyone got any suggestions?

    Thanks,

    Dave