Re: [Py4j-users] Multiprocessing error

[Eleftherios A. <ele...@df...>]

Hi Barthélémy,

thanks for your quick response. I am trying to understand your suggestion:
- On the Java side I have initialized GatewayServer on socket 25336
- On the Python side I initialized one GatewayClient per thread, 
connecting to socket 25336.

Is this the right solution so that the sockets are not being shared?

best
Eleftherios





On 22/06/12 15:55, Barthelemy Dagenais wrote:
> Hi,
>
> Please try to initialize the gateway in the thread/process and not outside. I believe you may be sharing sockets across threads/processes, which would explain the weird behavior you are observing.
>
> Barthélémy
>
> On 2012-06-22, at 9:08 AM, Eleftherios Avramidis wrote:
>
>> Dear all,
>>
>> I would like your help with a particular issue which occurs when I am
>> trying to use python multiprocessing, along with Py4j Java objects.
>>
>>
>> def singlethread(gateway):
>>      print "Thread starting"
>>
>>      #create a new view for the jvm
>>      meteor_view = gateway.new_jvm_view()
>>      #import required packages
>>      java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>>      #initialize the java object
>>      scorer = meteor_view.MeteorScorer()
>>      print "object initialized"
>>      #run object function
>>      stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
>>      print stats.score
>>
>> if __name__ == '__main__':
>>      socket_no = 25336
>>      gatewayclient = GatewayClient('localhost', socket_no)
>>      print "Gclient started"
>>      gateway = JavaGateway(gatewayclient, auto_convert=True,
>> auto_field=True)
>>      print "Gateway started"
>>
>>      p = multiprocessing.Pool(2)
>>      print "Multipool initialized"
>>      p.map(singlethread, [gateway, gateway, gateway])
>>
>>
>>
>> output:
>>
>> Gclient started
>> Gateway started
>> Multipool initialized
>> Exception in thread Thread-2:
>> Traceback (most recent call last):
>>    File "/usr/local/lib/python2.7/threading.py", line 552, in
>> __bootstrap_inner
>>      self.run()
>>    File "/usr/local/lib/python2.7/threading.py", line 505, in run
>>      self.__target(*self.__args, **self.__kwargs)
>>    File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 313, in
>> _handle_tasks
>>      put(task)
>>    File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py",
>> line 432, in __call__
>>      self.target_id, self.name)
>>    File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line
>> 271, in get_return_value
>>      raise Py4JError('An error occurred while calling %s%s%s.
>> Trace:\n%s\n' % (target_id, '.', name, value))
>> Py4JError: An error occurred while calling t.__getnewargs__. Trace:
>> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>>      at java.lang.String.substring(String.java:1949)
>>      at java.lang.String.substring(String.java:1916)
>>      at py4j.Gateway.invoke(Gateway.java:250)
>>      at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>>      at py4j.commands.CallCommand.execute(CallCommand.java:81)
>>      at py4j.GatewayConnection.run(GatewayConnection.java:175)
>>      at java.lang.Thread.run(Thread.java:636)
>>
>>
>> I would be happy if somebody can help me find out what exactly the error
>> may be, or if you can indicate where I should look for the problem.
>>
>> Interestingly enough, this problem doesn't occur when running without
>> multiprocessing. It also runs fine with other Java programs. But how can
>> I specify which aspect of the Java program causes the error?
>>
>> best
>> Eleftherios
>>
>> -- 
>> MSc. Inf. Eleftherios Avramidis
>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>> Tel. +49-30 238 95-1806
>>
>> Fax. +49-30 238 95-1810
>>
>> -------------------------------------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>> -------------------------------------------------------------------------------------------
>>
>>
>> ------------------------------------------------------------------------------
>> Live Security Virtual Conference
>> Exclusive live event will cover all the ways today's security and
>> threat landscape has changed and how IT managers can respond. Discussions
>> will include endpoint security, mobile security and the latest in malware
>> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>> _______________________________________________
>> Py4j-users mailing list
>> Py4...@li...
>> https://lists.sourceforge.net/lists/listinfo/py4j-users


-- 
MSc. Inf. Eleftherios Avramidis
DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
Tel. +49-30 238 95-1806

Fax. +49-30 238 95-1810

-------------------------------------------------------------------------------------------
Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern

Geschaeftsfuehrung:
Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
Dr. Walter Olthoff

Vorsitzender des Aufsichtsrats:
Prof. Dr. h.c. Hans A. Aukes

Amtsgericht Kaiserslautern, HRB 2313
-------------------------------------------------------------------------------------------