Menu
▾
▴
Re: [Jython-dev] Buffer protocol - direct vs. JVM ByteBuffers
|
From: Stefan R. <Ste...@gm...> - 2016-05-15 14:22:44
|
<html><head></head><body><div style="font-family: Verdana;font-size: 12.0px;"><div> <div> <div>Hey Jeff,</div> <div> </div> <div>A NIO ByteBuffer being direct makes the difference whether you can access it from native code via JNI:<br/> It provides access to the data in a direct NIO ByteBuffer via a C-style pointer. This means a native library can read or write the very same memory the buffer exposes on Java-side.<br/> So the support for the DIRECT_NIO flag would open up the whole world of JNI/C-use cases for Jython's buffer protocol.</div> <div> </div> <div>See</div> <div> </div> <div>http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html#nio_support</div> <div> </div> <div>for official documentation on this feature (i.e. related JNI-API).</div> <div><br/> My concrete use-case is that I can make JyNI put some sugar around this API such that it resembles CPython's buffer protocol, ultimately allowing to hand a Jython-style buffer protocol supporting PyObject from Jython to a CPython-style native C-extension which will recognize it as a CPython-style buffer protocol supporting object.</div> <div><br/> Honestly, direct NIO ByteBuffer is not the only solution to this. JNI can also offer a C-style memory pointer to a byte-array's data (under certain conditions). However this has some drawbacks, but might be suitable as a fallback to some extend.</div> <div> </div> <div>See</div> <div> </div> <div>http://docs.oracle.com/javase/7/docs/technotes/guides/jni/spec/functions.html#wp17314<br/> (scroll down to "Get<PrimitiveType>ArrayElements Routines")</div> <div> </div> <div>https://rkennke.wordpress.com/2007/07/28/efficient-jni-programming-iii-array-access</div> <div> </div> <div>Main issue here is that GetByteArrayElements does not guarantee to offer a view onto the data (i.e. if array pinning is not possible for whatever reason), but might only provide a copy. While modified data can be copied back, this would not be sufficient to support CPython's PyObject_GetBuffer method with the PyBUF_WRITABLE flag set.<br/> As a fallback I can (and maybe will) apply a copy-back operation on PyBuffer_Release call. This might work for some extensions, but I suppose the semantics of PyBUF_WRITABLE flag does not intend to have the write-operation only realized not before PyBuffer_Release call (CPython doc does not state this semantics explicitly though).</div> <div> </div> <div>To make the JVM try hard to provide the underlying array without copying I can use GetPrimitiveArrayCritical, but let me cite from its doc:</div> <div> </div> <div>"""<br/> However, there are significant restrictions on how these functions can be used.</div> <div>After calling GetPrimitiveArrayCritical, the native code should not run for an extended period of time before it calls ReleasePrimitiveArrayCritical. We must treat the code inside this pair of functions as running in a "critical region." Inside a critical region, native code must not call other JNI functions, or any system call that may cause the current thread to block and wait for another Java thread. (For example, the current thread must not call read on a stream being written by another Java thread.)</div> <div>These restrictions make it more likely that the native code will obtain an uncopied version of the array, even if the VM does not support pinning. For example, a VM may temporarily disable garbage collection when the native code is holding a pointer to an array obtained via GetPrimitiveArrayCritical.<br/> """</div> <div> </div> <div>Remember that the pointer would be passed to a native C-extension and that the extension's call to PyBuffer_Release would let JyNI trigger ReleasePrimitiveArrayCritical. Do I need to say that we must assume the native extension was designed to work with CPython and doesn't know or care about running in a "critical region" with the implied restrictions? So I think this fallback should only be applied to certain harmless extensions previously checked and white-listed, if at all.</div> <div> </div> <div>As I currently perceive it, direct NIO ByteBuffer is the only save and sufficient way for a full native BufferProtocol support.</div> <div> </div> <div>-Stefan</div> </div> <div> </div> <div> <div style="margin: 10.0px 5.0px 5.0px 10.0px;padding: 10.0px 0 10.0px 10.0px;border-left: 2.0px solid rgb(195,217,229);"> <div style="margin: 0 0 10.0px 0;"><b>Gesendet:</b> Sonntag, 15. Mai 2016 um 11:57 Uhr<br/> <b>Von:</b> "Jeff Allen" <ja...@fa...><br/> <b>An:</b> "Stefan Richthofer" <Ste...@gm...>, "Jython Developers" <jyt...@li...><br/> <b>Betreff:</b> [Jython-dev] Buffer protocol - direct vs. JVM ByteBuffers</div> <div> <div style="background-color: rgb(255,255,255);"> <p>Stefan:</p> <p><a class="moz-txt-link-freetext" href="https://github.com/jythontools/jython/pull/39" target="_blank">https://github.com/jythontools/jython/pull/39</a></p> <p>What difference does it make in your use case whether a NIO ByteBuffer is direct or non-direct? I can see why a client might want to know which it had been given, but not why it might want an exception raised in one or other case.</p> <p>Nothing I'm doing seems to depend on what kind of memory the exporting object has, therefore on the implementation type of <tt>ByteBuffer storage</tt>. It may depend on <tt>storage.hasArray()</tt>, but <tt>storage.isDirect()</tt> seems to make no difference.</p> Jeff <pre class="moz-signature">-- Jeff Allen</pre> ------------------------------------------------------------------------------ Mobile security can be enabling, not merely restricting. Employees who bring their own devices (BYOD) to work are irked by the imposition of MDM restrictions. Mobile Device Manager Plus allows you to control only the apps on BYO-devices by containerizing them, leaving personal data untouched! <a href="https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________" target="_blank">https://ad.doubleclick.net/ddm/clk/304595813;131938128;j_______________________________________________</a> Jython-dev mailing list Jyt...@li... <a href="https://lists.sourceforge.net/lists/listinfo/jython-dev" target="_blank">https://lists.sourceforge.net/lists/listinfo/jython-dev</a></div> </div> </div> </div> </div></div></body></html> |
