Re: [Py4j-users] Multiprocessing error

[Barthelemy D. <ba...@cs...>]

Hi,

Please try to initialize the gateway in the thread/process and not outside. I believe you may be sharing sockets across threads/processes, which would explain the weird behavior you are observing.

Barthélémy

On 2012-06-22, at 9:08 AM, Eleftherios Avramidis wrote:

> Dear all,
> 
> I would like your help with a particular issue which occurs when I am 
> trying to use python multiprocessing, along with Py4j Java objects.
> 
> 
> def singlethread(gateway):
>     print "Thread starting"
> 
>     #create a new view for the jvm
>     meteor_view = gateway.new_jvm_view()
>     #import required packages
>     java_import(meteor_view, 'edu.cmu.meteor.scorer.*')
>     #initialize the java object
>     scorer = meteor_view.MeteorScorer()
>     print "object initialized"
>     #run object function
>     stats = scorer.getMeteorStats("Test sentence", "Test sentence !");
>     print stats.score
> 
> if __name__ == '__main__':
>     socket_no = 25336
>     gatewayclient = GatewayClient('localhost', socket_no)
>     print "Gclient started"
>     gateway = JavaGateway(gatewayclient, auto_convert=True, 
> auto_field=True)
>     print "Gateway started"
> 
>     p = multiprocessing.Pool(2)
>     print "Multipool initialized"
>     p.map(singlethread, [gateway, gateway, gateway])
> 
> 
> 
> output:
> 
> Gclient started
> Gateway started
> Multipool initialized
> Exception in thread Thread-2:
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/threading.py", line 552, in 
> __bootstrap_inner
>     self.run()
>   File "/usr/local/lib/python2.7/threading.py", line 505, in run
>     self.__target(*self.__args, **self.__kwargs)
>   File "/usr/local/lib/python2.7/multiprocessing/pool.py", line 313, in 
> _handle_tasks
>     put(task)
>   File "/usr/local/lib/python2.7/site-packages/py4j/java_gateway.py", 
> line 432, in __call__
>     self.target_id, self.name)
>   File "/usr/local/lib/python2.7/site-packages/py4j/protocol.py", line 
> 271, in get_return_value
>     raise Py4JError('An error occurred while calling %s%s%s. 
> Trace:\n%s\n' % (target_id, '.', name, value))
> Py4JError: An error occurred while calling t.__getnewargs__. Trace:
> java.lang.StringIndexOutOfBoundsException: String index out of range: -1
>     at java.lang.String.substring(String.java:1949)
>     at java.lang.String.substring(String.java:1916)
>     at py4j.Gateway.invoke(Gateway.java:250)
>     at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:124)
>     at py4j.commands.CallCommand.execute(CallCommand.java:81)
>     at py4j.GatewayConnection.run(GatewayConnection.java:175)
>     at java.lang.Thread.run(Thread.java:636)
> 
> 
> I would be happy if somebody can help me find out what exactly the error 
> may be, or if you can indicate where I should look for the problem.
> 
> Interestingly enough, this problem doesn't occur when running without 
> multiprocessing. It also runs fine with other Java programs. But how can 
> I specify which aspect of the Java program causes the error?
> 
> best
> Eleftherios
> 
> -- 
> MSc. Inf. Eleftherios Avramidis
> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
> Tel. +49-30 238 95-1806
> 
> Fax. +49-30 238 95-1810
> 
> -------------------------------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
> 
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
> 
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
> 
> Amtsgericht Kaiserslautern, HRB 2313
> -------------------------------------------------------------------------------------------
> 
> 
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and 
> threat landscape has changed and how IT managers can respond. Discussions 
> will include endpoint security, mobile security and the latest in malware 
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> _______________________________________________
> Py4j-users mailing list
> Py4...@li...
> https://lists.sourceforge.net/lists/listinfo/py4j-users