Re: [Py4j-users] Multiprocessing error

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Eleftherios,

in this code:

if __name__ == '__main__':
    socket_no = 25336
    gatewayclient = GatewayClient('localhost', socket_no)
    print "Gclient started"
    gateway = JavaGateway(gatewayclient, auto_convert=True, 
auto_field=True)
    print "Gateway started"

    p = multiprocessing.Pool(2)
    print "Multipool initialized"
    p.map(singlethread, [gateway, gateway, gateway])

You are creating only one instance of JavaGateway and this instance is passed to each worker (the list [gateway, gateway, gateway] is in fact a list of three pointers pointing to the same instance). I usually recommend to create one JavaGateway instance per worker.

Unless this was not the code that caused the problem, I would start by creating a JavaGateway in each worker and then, if you still encounter the problem, we can start looking at other possibilities.

Thanks,
Barthélémy

On 2012-06-22, at 10:20 AM, Eleftherios Avramidis wrote:

> Hi Barthélémy,
> 
> thanks for your quick response. I am trying to understand your suggestion:
> - On the Java side I have initialized GatewayServer on socket 25336
> - On the Python side I initialized one GatewayClient per thread, 
> connecting to socket 25336.
> 
> Is this the right solution so that the sockets are not being shared?
> 
> best
> Eleftherios
> 
> 
> 
> 
> 
> On 22/06/12 15:55, Barthelemy Dagenais wrote:
>> Hi,
>> 
>> Please try to initialize the gateway in the thread/process and not outside. I believe you may be sharing sockets across threads/processes, which would explain the weird behavior you are observing.
>> 
>> Barthélémy
>> 
>> On 2012-06-22, at 9:08 AM, Eleftherios Avramidis wrote:
>> 
>>> Dear all,
>>> 
>>> I would like your help with a particular issue which occurs when I am
>>> trying to use python multiprocessing, along with Py4j Java objects.
>>> 
>>> 
>>> def singlethread(gateway):
>>>     print "Thread starting"
>>> 
>>>     #create a new view for the jvm
>>>     meteor_view = gateway.new_jvm_view()
>>>     #import required packages
>>>     java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>>>     #initialize the java object
>>>     scorer = meteor_view.MeteorScorer()
>>>     print "object initialized"
>>>     #run object function
>>>     stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
>>>     print stats.score
>>> 
>>> if __name__ == '__main__':
>>>     socket_no = 25336
>>>     gatewayclient = GatewayClient('localhost', socket_no)
>>>     print "Gclient started"
>>>     gateway = JavaGateway(gatewayclient, auto_convert=True,
>>> auto_field=True)
>>>     print "Gateway started"
>>> 
>>>     p = multiprocessing.Pool(2)
>>>     print "Multipool initialized"
>>>     p.map(singlethread, [gateway, gateway, gateway])
>>> 
>>> 
>>> 
>>> output:
>>> 
>>> Gclient started
>>> Gateway started
>>> Multipool initialized
>>> Exception in thread Thread-2:
>>> Traceback (most recent call last):
>>>   File "/usr/local/lib/python2.7/threading.py", line 552, in
>>> __bootstrap_inner
>>>     self.run()
>>>   File "/usr/local/lib/python2.7/threading.py", line 505, in run
>>>     self.__target(*self.__args, **self.__kwargs)
>>>   File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 313, in
>>> _handle_tasks
>>>     put(task)
>>>   File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py",
>>> line 432, in __call__
>>>     self.target_id, self.name)
>>>   File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line
>>> 271, in get_return_value
>>>     raise Py4JError('An error occurred while calling %s%s%s.
>>> Trace:\n%s\n' % (target_id, '.', name, value))
>>> Py4JError: An error occurred while calling t.__getnewargs__. Trace:
>>> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>>>     at java.lang.String.substring(String.java:1949)
>>>     at java.lang.String.substring(String.java:1916)
>>>     at py4j.Gateway.invoke(Gateway.java:250)
>>>     at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>>>     at py4j.commands.CallCommand.execute(CallCommand.java:81)
>>>     at py4j.GatewayConnection.run(GatewayConnection.java:175)
>>>     at java.lang.Thread.run(Thread.java:636)
>>> 
>>> 
>>> I would be happy if somebody can help me find out what exactly the error
>>> may be, or if you can indicate where I should look for the problem.
>>> 
>>> Interestingly enough, this problem doesn't occur when running without
>>> multiprocessing. It also runs fine with other Java programs. But how can
>>> I specify which aspect of the Java program causes the error?
>>> 
>>> best
>>> Eleftherios
>>> 
>>> -- 
>>> MSc. Inf. Eleftherios Avramidis
>>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>>> Tel. +49-30 238 95-1806
>>> 
>>> Fax. +49-30 238 95-1810
>>> 
>>> -------------------------------------------------------------------------------------------
>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>> 
>>> Geschaeftsfuehrung:
>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>> Dr. Walter Olthoff
>>> 
>>> Vorsitzender des Aufsichtsrats:
>>> Prof. Dr. h.c. Hans A. Aukes
>>> 
>>> Amtsgericht Kaiserslautern, HRB 2313
>>> -------------------------------------------------------------------------------------------
>>> 
>>> 
>>> ------------------------------------------------------------------------------
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. Discussions
>>> will include endpoint security, mobile security and the latest in malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Py4j-users mailing list
>>> Py4...@li...
>>> https://lists.sourceforge.net/lists/listinfo/py4j-users
> 
> 
> -- 
> MSc. Inf. Eleftherios Avramidis
> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
> Tel. +49-30 238 95-1806
> 
> Fax. +49-30 238 95-1810
> 
> -------------------------------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
> 
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
> 
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
> 
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Py4j-users mailing list
> Py4...@li...
> https://lists.sourceforge.net/lists/listinfo/py4j-users