Dave Rigby - 2014-02-10

I'm trying to use multiprocessing.pool to speed up some parsing of a file parsed using pyparsing, however I get a multiprocessing.pool.MaybeEncodingError exception whenever I try this.

I've narrowed it down to something to do with returning a dictionary (ParseResults.asDict()), using asList() the error doesn't occur; but the input I'm actually parsing is pretty complex so ideally I'd like to use asDict.

Simplified testcase:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
#!/usr/bin/env python2.7
from pyparsing import *
import multiprocessing

value = ( Word(alphas) )
key_val = Group( value + ":" + value )
lang = Dict( Suppress('[') + delimitedList( key_val ) + Suppress(']') )

def parse_dict(s):
    return lang.parseString(s).asDict()

def parse_list(s):
    return lang.parseString(s).asList()

data = ['[ foo : bar ]']
pool = multiprocessing.Pool()

# This works (list)
pool.map(parse_list, data)

# This fails (dict)
pool.map(parse_dict, data)

Fails with:

Traceback (most recent call last):
  File "lib/python/nutshell/multi_parse.py", line 19, in <module>
    pool.map(parse, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 250, in map
    return self.map_async(func, iterable, chunksize).get()
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/pool.py", line 554, in get
    raise self._value
multiprocessing.pool.MaybeEncodingError: Error sending result: '[{'foo': ([':', 'bar'], {})}]'. Reason: 'TypeError("'str' object is not callable",)'

Anyone got any suggestions?

Thanks,

Dave