Re: [Py4j-users] faster way to convert arrays from python to Java

[Barthelemy D. <bar...@in...>]

I'll probably be able to speed up the transfer of a byte arrays between
Java and Python by just passing the length of the byte array and the actual
bytes through the socket connection between the JVM and the Python process.
Currently, these bytes are converted to and from BASE64 which slows
everything but keeps the protocol easier to work with.

I'm not sure if this will help your use case though. Does the BoofCV
library accepts byte[]? Is it possible/desirable to transform your numpy
structure into a byte array?

We could use memory mapped files to exchange data between the two
processes, but support in Python is tricky because you need to know in
advance if you are on UNIX or Windows. I'm not even sure that it would be
faster than just transferring everything through the socket connection.

Barthelemy



On Wed, Jul 15, 2015 at 8:33 PM Peter <dih...@gm...> wrote:

> I know this is a known/fundamental issue, but just wanted place my request
> for a faster way to convert large arrays between the two.  I'm written a
> wrapper around BoofCV, which is a computer vision library in Java, using
> Py4J.  For the most part it has been really easy to do and works well.  The
> one major problem I have is that loading an image in python then passing it
> to Java is glacial.
>
> For example, when tracking objects inside a video sequence it takes 28ms
> to process the image, but 95ms to convert it from a numpy format into the
> BoofCV format!  Kinda kills any hope of doing something in real-time.  Yes
> I know you said not to use this library if you want speed, but it's
> actually almost there.
>
> Would it be possible to do some sort of hackery like share a large chunk
> of memory between the two and use that to do copies?
>
> - Peter
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> Py4j-users mailing list
> Py4...@li...
> https://lists.sourceforge.net/lists/listinfo/py4j-users
>