Re: [Py4j-users] Multiprocessing error

[Eleftherios A. <ele...@df...>]

Hi,

I am sorry, I missed two lines:


from py4j.java_gateway import JavaGateway, GatewayClient, java_import

import multiprocessing, time, random

def singlethread(socket_no):
     print "Thread starting"

     gatewayclient = GatewayClient('localhost', socket_no)
     gateway = JavaGateway(gatewayclient, auto_convert=True, 
auto_field=True)
     #create a new view for the jvm
     meteor_view = gateway.new_jvm_view()
     #import required packages
     java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
     #initialize the java object
     java_import(meteor_view, 'edu.cmu.meteor.util.*')
     #pass the language setting into the meteor configuration object
     config = meteor_view.MeteorConfiguration();
     config.setLanguage("en");
     scorer = meteor_view.MeteorScorer(config)
     print "object initialized"
     #run object function
     stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
     print stats.score
     return 1


if __name__ == '__main__':
     socket_no = 25336
     print "Gclient started"

     p = multiprocessing.Pool(3)
     print "Multipool initialized"
     p.map(singlethread, [socket_no, socket_no,socket_no])




On 22/06/12 16:50, Eleftherios Avramidis wrote:
> Hi Barthélémy,
>
> I just tried rewriting the code as following. Two threads printed out 
> their result successfully, the third pool worker gave a similar error. 
> (I am running this in a 2-core machine)
>
>
> from py4j.java_gateway import JavaGateway, GatewayClient, java_import
>
> import multiprocessing, time, random
>
> def singlethread(socket_no):
>     print "Thread starting"
>         gatewayclient = GatewayClient('localhost', socket_no)
>     gateway = JavaGateway(gatewayclient, auto_convert=True, 
> auto_field=True)
>     #create a new view for the jvm
>     meteor_view = gateway.new_jvm_view()
>     #import required packages
>     java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>     #initialize the java object
>     java_import(meteor_view, 'edu.cmu.meteor.util.*')
>     #pass the language setting into the meteor configuration object
>     config = meteor_view.MeteorConfiguration();
>     config.setLanguage("en");
>     scorer = meteor_view.MeteorScorer(config)
>     print "object initialized"
>     #run object function
>     stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
>     print stats.score
>     return 1
>
>
> if __name__ == '__main__':
>     socket_no = 25336
> #    gatewayclient = GatewayClient('localhost', socket_no)
>     print "Gclient started"
>
>
>
> output:
>
> Gclient started
> Multipool initialized
> Thread starting
> Thread starting
> Thread starting
> Process PoolWorker-3:
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 
> 258, in _bootstrap
>     self.run()
>   File "/usr/local/lib/python2.7/multiprocessing/process.py", line 
> 114, in run
>     self._target(*self._args, **self._kwargs)
>   File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 99, in 
> worker
>     put((job, i, result))
>   File "/usr/local/lib/python2.7/multiprocessing/queues.py", line 392, 
> in put
>     return send(obj)
>   File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", 
> line 432, in __call__
>     self.target_id, self.name)
>   File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line 
> 271, in get_return_value
>     raise Py4JError('An error occurred while calling %s%s%s. 
> Trace:\n%s\n' % (target_id, '.', name, value))
> Py4JError: An error occurred while calling o6.__getnewargs__. Trace:
> py4j.Py4JException: Method __getnewargs__([]) does not exist
>     at 
> py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:346)
>     at 
> py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:355)
>     at py4j.Gateway.invoke(Gateway.java:247)
>     at 
> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>     at py4j.commands.CallCommand.execute(CallCommand.java:81)
>     at py4j.GatewayConnection.run(GatewayConnection.java:175)
>     at java.lang.Thread.run(Thread.java:636)
>
>
> object initialized
> 0.335206780367
> object initialized
> 0.335206780367
>
>
> Any suggestions welcome,
>
> Lefteris
>
>
> On 22/06/12 16:25, Barthelemy Dagenais wrote:
>> Hi Eleftherios,
>>
>> in this code:
>>
>> if __name__ == '__main__':
>>      socket_no = 25336
>>      gatewayclient = GatewayClient('localhost', socket_no)
>>      print "Gclient started"
>>      gateway = JavaGateway(gatewayclient, auto_convert=True,
>> auto_field=True)
>>      print "Gateway started"
>>
>>      p = multiprocessing.Pool(2)
>>      print "Multipool initialized"
>>      p.map(singlethread, [gateway, gateway, gateway])
>>
>> You are creating only one instance of JavaGateway and this instance 
>> is passed to each worker (the list [gateway, gateway, gateway] is in 
>> fact a list of three pointers pointing to the same instance). I 
>> usually recommend to create one JavaGateway instance per worker.
>>
>> Unless this was not the code that caused the problem, I would start 
>> by creating a JavaGateway in each worker and then, if you still 
>> encounter the problem, we can start looking at other possibilities.
>>
>> Thanks,
>> Barthélémy
>>
>>
>> On 2012-06-22, at 10:20 AM, Eleftherios Avramidis wrote:
>>
>>> Hi Barthélémy,
>>>
>>> thanks for your quick response. I am trying to understand your 
>>> suggestion:
>>> - On the Java side I have initialized GatewayServer on socket 25336
>>> - On the Python side I initialized one GatewayClient per thread,
>>> connecting to socket 25336.
>>>
>>> Is this the right solution so that the sockets are not being shared?
>>>
>>> best
>>> Eleftherios
>>>
>>>
>>>
>>>
>>>
>>> On 22/06/12 15:55, Barthelemy Dagenais wrote:
>>>> Hi,
>>>>
>>>> Please try to initialize the gateway in the thread/process and not 
>>>> outside. I believe you may be sharing sockets across 
>>>> threads/processes, which would explain the weird behavior you are 
>>>> observing.
>>>>
>>>> Barthélémy
>>>>
>>>> On 2012-06-22, at 9:08 AM, Eleftherios Avramidis wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I would like your help with a particular issue which occurs when I am
>>>>> trying to use python multiprocessing, along with Py4j Java objects.
>>>>>
>>>>>
>>>>> def singlethread(gateway):
>>>>>      print "Thread starting"
>>>>>
>>>>>      #create a new view for the jvm
>>>>>      meteor_view = gateway.new_jvm_view()
>>>>>      #import required packages
>>>>>      java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>>>>>      #initialize the java object
>>>>>      scorer = meteor_view.MeteorScorer()
>>>>>      print "object initialized"
>>>>>      #run object function
>>>>>      stats = scorer.getMeteorStats("Test sentence", "Test sentence 
>>>>> !");
>>>>>      print stats.score
>>>>>
>>>>> if __name__ == '__main__':
>>>>>      socket_no = 25336
>>>>>      gatewayclient = GatewayClient('localhost', socket_no)
>>>>>      print "Gclient started"
>>>>>      gateway = JavaGateway(gatewayclient, auto_convert=True,
>>>>> auto_field=True)
>>>>>      print "Gateway started"
>>>>>
>>>>>      p = multiprocessing.Pool(2)
>>>>>      print "Multipool initialized"
>>>>>      p.map(singlethread, [gateway, gateway, gateway])
>>>>>
>>>>>
>>>>>
>>>>> output:
>>>>>
>>>>> Gclient started
>>>>> Gateway started
>>>>> Multipool initialized
>>>>> Exception in thread Thread-2:
>>>>> Traceback (most recent call last):
>>>>>    File "/usr/local/lib/python2.7/threading.py", line 552, in
>>>>> __bootstrap_inner
>>>>>      self.run()
>>>>>    File "/usr/local/lib/python2.7/threading.py", line 505, in run
>>>>>      self.__target(*self.__args, **self.__kwargs)
>>>>>    File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 
>>>>> 313, in
>>>>> _handle_tasks
>>>>>      put(task)
>>>>>    File 
>>>>> "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py",
>>>>> line 432, in __call__
>>>>>      self.target_id, self.name)
>>>>>    File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", 
>>>>> line
>>>>> 271, in get_return_value
>>>>>      raise Py4JError('An error occurred while calling %s%s%s.
>>>>> Trace:\n%s\n' % (target_id, '.', name, value))
>>>>> Py4JError: An error occurred while calling t.__getnewargs__. Trace:
>>>>> java.lang.StringIndexOutOfBoundsException: String index out of 
>>>>> range: -1
>>>>>      at java.lang.String.substring(String.java:1949)
>>>>>      at java.lang.String.substring(String.java:1916)
>>>>>      at py4j.Gateway.invoke(Gateway.java:250)
>>>>>      at 
>>>>> py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>>>>>      at py4j.commands.CallCommand.execute(CallCommand.java:81)
>>>>>      at py4j.GatewayConnection.run(GatewayConnection.java:175)
>>>>>      at java.lang.Thread.run(Thread.java:636)
>>>>>
>>>>>
>>>>> I would be happy if somebody can help me find out what exactly the 
>>>>> error
>>>>> may be, or if you can indicate where I should look for the problem.
>>>>>
>>>>> Interestingly enough, this problem doesn't occur when running without
>>>>> multiprocessing. It also runs fine with other Java programs. But 
>>>>> how can
>>>>> I specify which aspect of the Java program causes the error?
>>>>>
>>>>> best
>>>>> Eleftherios
>>>>>
>>>>> -- 
>>>>> MSc. Inf. Eleftherios Avramidis
>>>>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>>>>> Tel. +49-30 238 95-1806
>>>>>
>>>>> Fax. +49-30 238 95-1810
>>>>>
>>>>> ------------------------------------------------------------------------------------------- 
>>>>>
>>>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>>>
>>>>> Geschaeftsfuehrung:
>>>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>>>> Dr. Walter Olthoff
>>>>>
>>>>> Vorsitzender des Aufsichtsrats:
>>>>> Prof. Dr. h.c. Hans A. Aukes
>>>>>
>>>>> Amtsgericht Kaiserslautern, HRB 2313
>>>>> ------------------------------------------------------------------------------------------- 
>>>>>
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------ 
>>>>>
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond. 
>>>>> Discussions
>>>>> will include endpoint security, mobile security and the latest in 
>>>>> malware
>>>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>>>> _______________________________________________
>>>>> Py4j-users mailing list
>>>>> Py4...@li...
>>>>> https://lists.sourceforge.net/lists/listinfo/py4j-users
>>>
>>> -- 
>>> MSc. Inf. Eleftherios Avramidis
>>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>>> Tel. +49-30 238 95-1806
>>>
>>> Fax. +49-30 238 95-1810
>>>
>>> ------------------------------------------------------------------------------------------- 
>>>
>>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>>
>>> Geschaeftsfuehrung:
>>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>>> Dr. Walter Olthoff
>>>
>>> Vorsitzender des Aufsichtsrats:
>>> Prof. Dr. h.c. Hans A. Aukes
>>>
>>> Amtsgericht Kaiserslautern, HRB 2313
>>> ------------------------------------------------------------------------------------------- 
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------ 
>>>
>>> Live Security Virtual Conference
>>> Exclusive live event will cover all the ways today's security and
>>> threat landscape has changed and how IT managers can respond. 
>>> Discussions
>>> will include endpoint security, mobile security and the latest in 
>>> malware
>>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>> _______________________________________________
>>> Py4j-users mailing list
>>> Py4...@li...
>>> https://lists.sourceforge.net/lists/listinfo/py4j-users
>
>


-- 
MSc. Inf. Eleftherios Avramidis
DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
Tel. +49-30 238 95-1806

Fax. +49-30 238 95-1810

-------------------------------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------------------------------------