Re: [Py4j-users] Multiprocessing error

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Hi Barthélémy,

I just tried rewriting the code as following. Two threads printed out their result successfully, the third pool worker gave a similar error. (I am running this in a 2-core machine)

from py4j.java_gateway import JavaGateway, GatewayClient, java_import

import multiprocessing, time, random

def singlethread(socket_no):
     print "Thread starting"

     gatewayclient = GatewayClient('localhost', socket_no)
     gateway = JavaGateway(gatewayclient, auto_convert=True, auto_field=True)
     #create a new view for the jvm
     meteor_view = gateway.new_jvm_view()
     #import required packages
     java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
     #initialize the java object
     java_import(meteor_view, 'edu.cmu.meteor.util.*')
     #pass the language setting into the meteor configuration object
     config = meteor_view.MeteorConfiguration();
     config.setLanguage("en");
     scorer = meteor_view.MeteorScorer(config)
     print "object initialized"
     #run object function
     stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
     print stats.score
     return 1

if __name__ == '__main__':
     socket_no = 25336
#    gatewayclient = GatewayClient('localhost', socket_no)
     print "Gclient started"

output:

Gclient started
Multipool initialized
Thread starting
Thread starting
Thread starting
Process PoolWorker-3:
Traceback (most recent call last):
   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 258, 
in _bootstrap
     self.run()
   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 114, 
in run
     self._target(*self._args, **self._kwargs)
   File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 99, in 
worker
     put((job, i, result))
   File "/usr/local/lib/python2.7/multiprocessing/queues.py", line 392, 
in put
     return send(obj)
   File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", 
line 432, in __call__
     self.target_id, self.name)
   File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line 
271, in get_return_value
     raise Py4JError('An error occurred while calling %s%s%s. 
Trace:\n%s\n' % (target_id, '.', name, value))
Py4JError: An error occurred while calling o6.__getnewargs__. Trace:
py4j.Py4JException: Method __getnewargs__([]) does not exist
     at 
py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:346)
     at 
py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:355)
     at py4j.Gateway.invoke(Gateway.java:247)
     at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
     at py4j.commands.CallCommand.execute(CallCommand.java:81)
     at py4j.GatewayConnection.run(GatewayConnection.java:175)
     at java.lang.Thread.run(Thread.java:636)

object initialized
0.335206780367
object initialized
0.335206780367

Any suggestions welcome,

Lefteris

On 22/06/12 16:25, Barthelemy Dagenais wrote:
> Hi Eleftherios,
>
> in this code:
>
> if __name__ == '__main__':
>      socket_no = 25336
>      gatewayclient = GatewayClient('localhost', socket_no)
>      print "Gclient started"
>      gateway = JavaGateway(gatewayclient, auto_convert=True,
> auto_field=True)
>      print "Gateway started"
>
>      p = multiprocessing.Pool(2)
>      print "Multipool initialized"
>      p.map(singlethread, [gateway, gateway, gateway])
>
> You are creating only one instance of JavaGateway and this instance is passed to each worker (the list [gateway, gateway, gateway] is in fact a list of three pointers pointing to the same instance). I usually recommend to create one JavaGateway instance per worker.
>
> Unless this was not the code that caused the problem, I would start by creating a JavaGateway in each worker and then, if you still encounter the problem, we can start looking at other possibilities.
>
> Thanks,
> Barthélémy
>
>
> On 2012-06-22, at 10:20 AM, Eleftherios Avramidis wrote:
>
>> Hi Barthélémy,
>>
>> thanks for your quick response. I am trying to understand your suggestion:
>> - On the Java side I have initialized GatewayServer on socket 25336
>> - On the Python side I initialized one GatewayClient per thread,
>> connecting to socket 25336.
>>
>> Is this the right solution so that the sockets are not being shared?
>>
>> best
>> Eleftherios
>>
>>
>>
>>
>>
>> On 22/06/12 15:55, Barthelemy Dagenais wrote:
>>> Hi,
>>>
>>> Please try to initialize the gateway in the thread/process and not outside. I believe you may be sharing sockets across threads/processes, which would explain the weird behavior you are observing.
>>>
>>> Barthélémy
>>>
>>> On 2012-06-22, at 9:08 AM, Eleftherios Avramidis wrote:
>>>
>>>> Dear all,
>>>>
>>>> I would like your help with a particular issue which occurs when I am
>>>> trying to use python multiprocessing, along with Py4j Java objects.
>>>>
>>>>
>>>> def singlethread(gateway):
>>>>      print "Thread starting"
>>>>
>>>>      #create a new view for the jvm
>>>>      meteor_view = gateway.new_jvm_view()
>>>>      #import required packages
>>>>      java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>>>>      #initialize the java object
>>>>      scorer = meteor_view.MeteorScorer()
>>>>      print "object initialized"
>>>>      #run object function
>>>>      stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
>>>>      print stats.score
>>>>
>>>> if __name__ == '__main__':
>>>>      socket_no = 25336
>>>>      gatewayclient = GatewayClient('localhost', socket_no)
>>>>      print "Gclient started"
>>>>      gateway = JavaGateway(gatewayclient, auto_convert=True,
>>>> auto_field=True)
>>>>      print "Gateway started"
>>>>
>>>>      p = multiprocessing.Pool(2)
>>>>      print "Multipool initialized"
>>>>      p.map(singlethread, [gateway, gateway, gateway])
>>>>
>>>>
>>>>
>>>> output:
>>>>
>>>> Gclient started
>>>> Gateway started
>>>> Multipool initialized
>>>> Exception in thread Thread-2:
>>>> Traceback (most recent call last):
>>>>    File "/usr/local/lib/python2.7/threading.py", line 552, in
>>>> __bootstrap_inner
>>>>      self.run()
>>>>    File "/usr/local/lib/python2.7/threading.py", line 505, in run
>>>>      self.__target(*self.__args, **self.__kwargs)
>>>>    File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 313, in
>>>> _handle_tasks
>>>>      put(task)
>>>>    File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py",
>>>> line 432, in __call__
>>>>      self.target_id, self.name)
>>>>    File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line
>>>> 271, in get_return_value
>>>>      raise Py4JError('An error occurred while calling %s%s%s.
>>>> Trace:\n%s\n' % (target_id, '.', name, value))
>>>> Py4JError: An error occurred while calling t.__getnewargs__. Trace:
>>>> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>>>>      at java.lang.String.substring(String.java:1949)
>>>>      at java.lang.String.substring(String.java:1916)
>>>>      at py4j.Gateway.invoke(Gateway.java:250)
>>>>      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>>>>      at py4j.commands.CallCommand.execute(CallCommand.java:81)
>>>>      at py4j.GatewayConnection.run(GatewayConnection.java:175)
>>>>      at java.lang.Thread.run(Thread.java:636)
>>>>
>>>>
>>>> I would be happy if somebody can help me find out what exactly the error
>>>> may be, or if you can indicate where I should look for the problem.
>>>>
>>>> Interestingly enough, this problem doesn't occur when running without
>>>> multiprocessing. It also runs fine with other Java programs. But how can
>>>> I specify which aspect of the Java program causes the error?
>>>>
>>>> best
>>>> Eleftherios
>>>>
>>>> -- 
>>>> MSc. Inf. Eleftherios Avramidis
>>>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>>>> Tel. +49-30 238 95-1806
>>>>
>>>> Fax. +49-30 238 95-1810
>>>>
>>>> -------------------------------------------------------------------------------------------
>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>
>>>> Geschaeftsfuehrung:
>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>> Dr. Walter Olthoff
>>>>
>>>> Vorsitzender des Aufsichtsrats:
>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>
>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>> -------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Live Security Virtual Conference
>>>> Exclusive live event will cover all the ways today's security and
>>>> threat landscape has changed and how IT managers can respond. Discussions
>>>> will include endpoint security, mobile security and the latest in malware
>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>> _______________________________________________
>>>> Py4j-users mailing list
>>>> Py4...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/py4j-users
>>
>> -- 
>> MSc. Inf. Eleftherios Avramidis
>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>> Tel. +49-30 238 95-1806
>>
>> Fax. +49-30 238 95-1810
>>
>> -------------------------------------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Py4j-users mailing list
>> Py4...@li...
>> https://lists.sourceforge.net/lists/listinfo/py4j-users

-- 
MSc. Inf. Eleftherios Avramidis
DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
Tel. +49-30 238 95-1806

Fax. +49-30 238 95-1810

-------------------------------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------------------------------------