Re: [Jython-dev] Jython buffer protocol

[Jim B. <jim...@py...>]

On Wed, Apr 27, 2016 at 4:36 PM, Jeff Allen <ja...@fa...> wrote:

> I'm giving serious consideration to idea 2, that is, the storage
> implementation is j.n.ByteBuffer, always, and *may* wrap a byte[] object.
> I'd need to try this out to ensure there is no fatal flaw.
>
> *Jim:* this is a breaking change to the API. Do we need to be more careful
> of possible users? I suspect we are only breaking our own work here: how
> about you?
>
We should mention such a breaking change. Necessarily we have been very
conservative on various aspects of our Java API - there is certainly usage
out there. But that has been seen in 2.5 or earlier API definitions. I
don't see a problem here - any users will be sophisticated and can readily
adapt.

> We would be saying in this that the Jython PyBuffer is allowed to be less
> like the CPython one than I've been aiming for. This consistency may be
> less important than Stefan's use case. The CPython protocol promises
> efficient access to the storage of an object via a pointer, and we would be
> saying "only as efficient as a j.n.ByteBuffer" ... although it may turn out
> there's a backing array. j.n.ByteBuffer does not replace PyBuffer, because
> it cannot describe strided access or the get-release behaviour.
>
> I think this leads to an API in which what I've tried to do with
> PyBuffer.Pointer we now do by handing out ByteBuffer slices. So Pointer
> goes away. In that case getBuf() and getNIOByteBuffer() are probably the
> same thing. I do not think it is safe to hand out the actual storage: it is
> almost unavoidable clients would manipulate the internal state (position,
> limit), surprising each other and the PyBuffer implementation if it relies
> on them, as I think it should.
>
> Concerning the pointer to object member in CPython Py_Buffer, it seems to
> be a 3.x feature, which may be why I haven't replicated it. It seems easy
> to add. (I'd be rewriting all the constructors anyway.) In CPython it's
> null when there's a buffer but no object.
>
> Jeff Allen
>
> On 24/04/2016 15:36, Stefan Richthofer wrote:
>
> Jeff,
>
> good to hear that you can help with this stuff and also that your answer implies you don't have concerns with the new feature itself. Thinking it through again, I think the following way would be cleanest to add this functionality:
>
> Add a ByteBuffer-type storage, either exclusively or in addition to byte[] storage.
>
>
>
> 1) Version with additional field java.nio.ByteBuffer bufferStorage:
>
> Case byte[]-backed PyBuffer:
> (buffer storage must be view on storage, i.e. backed by it and must always point to first element)
>
> storage is byte[]
> bufferStorage is ByteBuffer.wrap(storage)
>
> getNIOByteBuffer() can use bufferStorage and needn't call ByteBuffer.wrap every time again.
>
>
> Case direct ByteBuffer (likely not having backing array):
>
> storage is null or if the JVM happens to be capable of providing direct ByteBuffer with byte[] backend: bufferStorage.array()
>
> bufferStorage is ByteBuffer.allocateDirect(capacity)
>
> Methods that used to access elements of storage directly are enriched by a fallback for case storage == null. The fallback would directly operate on bufferStorage.
>
>
>
>
> 2) Version with exclusive Buffer-storage:
>
> storage type is java.nioByteBuffer instead of byte[]
>
>
> Case byte[]-backed PyBuffer:
>
> storage is ByteBuffer.allocate(capacity) (i.e. non-Direct, so buffer will have backing array!)
>
> getNIOByteBuffer() can use storage and needn't call ByteBuffer.wrap.
>
> Methods that used to access elements of storage directly now do this on storage.array() rather than on storage itself (should be doable by a simple search/replace refactoring more or less).
>
>
> Case direct ByteBuffer (likely not having backing array):
>
> bufferStorage is ByteBuffer.allocateDirect(capacity)
>
> Methods that used to access elements of storage directly are enriched by a fallback for case storage.hasArray() == false. The fallback would directly operate on storage's ByteBuffer methods.
>
>
> I can do the work of writing the fallbacks or help with it up to your discretion.
>
>
> Then another thing: I noticed CPython's PyBuffer-pendant contains a reference to the PyObject that exported it, so you can always find the origin of a given PyBuffer. I don't see how this would be feasible with Jython's current PyBuffer implementation. So from JyNI perspective I can store (as a mapping) the exporter in case it is known for some reason, e.g. because PyBuffer was converted from a native CPython-like variant.
> However there could be situations where the buffer comes from Jython and the origin would be unknown. In that case I would (currently) just provide a NULL-value or PyNone for this field and hope to get away with it for the important extensions. Maybe we could attach a PyBuffer's origin in Jython too...? (e.g. as a JyAttribute only if some global flag is set, which JyNI would then set on load).
>
> Best
>
> Stefan
>
>
>
>
> Gesendet: Samstag, 23. April 2016 um 20:14 Uhr
> Von: "Jeff Allen" <ja...@fa...> <ja...@fa...>
> An: "Stefan Richthofer" <Ste...@gm...> <Ste...@gm...>
> Cc: jim...@py...
> Betreff: Re: [Jython-dev] Jython buffer protocol
>
> Hi Stefan.
>
> Refreshing my memory about how these classes work, I can see that I took
> at face value the CPython view that the purpose of the buffer interface
> is to give clients access to the underlying array of bytes, so
> abstraction of the storage always gave way to what I thought would be
> efficient. (Abstraction of the unit to be something other than byte is
> sketched but clarity and a use case eluded me.)
>
> I always feel I've failed if I have to cast. My instinct is for option a.
>
> But I think you would not create a "Direct" parallel to BaseBuffer,
> since it contains a lot of helper methods independent of the storage
> implementation. Rather, factor it into two layers, the first being
> either BaseBuffer or AbstractBuffer (depending on what causes least
> pain) and the next layer being two base classes, one the revised
> BaseBuffer containing:
>      protected byte[] storage;
> and the other containing:
>      protected ByteBuffer storage;
> And in each you migrate case whatever it seems natural should come along
> with these declarations.
>
> I've been meaning to get back to Jython: I could do this groundwork if
> that would not be confusing.
>
> Jeff
>
> Jeff Allen
>
> On 22/04/2016 21:50, Stefan Richthofer wrote:
>
> Hello Jeff,
>
> I'm warming up this old thread, because I am about to start actual work on JyNI's support
> for buffer-protocol / the PyBuffer builtin type.
> I'd like to point you to my recent pull request https://github.com/jythontools/jython/pull/39.
> It's a preliminary step for adding support for direct java.nio.ByteBuffers. After establishing this flag
> I am going to add some actual support for it. I see basically two ways to go for this
>
> a) Create a parallel class hierarchy to BaseBuffer et al, backed by direct ByteBuffers. E.g.
> call everything with "Direct": DirectBaseBuffer, DirectSimpleBuffer etc.
> Then let BufferProtocol implementers check for the flag and use Direct counterpart of the
> usually used Buffer-Class accordingly.
>
> or
>
> b) Modify existing BaseBuffer such that storage is Object rather than byte[]. Then according to
> flags it will be byte[] or ByteBuffer. This variant will involve more explicit type casting than
> a), but would involve fewer new classes however.
>
> What is your opinion about this?
>
> Best
>
> Stefan
>
>
>
> Gesendet: Donnerstag, 07. August 2014 um 21:49 Uhr
> Von: "Jeff Allen" <ja...@fa...> <ja...@fa...>
> An: "Stefan Richthofer" <Ste...@gm...> <Ste...@gm...>
> Cc: "Jython Developers" <jyt...@li...> <jyt...@li...>
> Betreff: Re: [Jython-dev] Jython buffer protocol
>
>
> Jeff Allen
>
> On 06/08/2014 02:28, Stefan Richthofer wrote:
>
> Jeff,
>
> I just quickly scanned the changes and everything looks fine as far as I see. PyByteArray and BaseBytes may need adjustments too (on this occasion remember to add resizeCheck() in irepeat).
>
> Thanks for looking that over. I was rather asking whether the use case
> was served adequately by the addition of PyBUF.AS_ARRAY, hasArray() and
> getNIOByteBuffer(), since you obviously have a clear idea of it.
>
> I took a second/third look but didn't find anything to change in
> BaseBytes and PyByteArray: when arguments objects have the Buffer API
> they are accessed through the abstract API (not as byte[]). PyByteArray
> obviously can support AS_ARRAY, and that seems to be covered in
> SimpleBuffer.
>
> I fixed the irepeat bug in the previous change set: it's a distinct
> issue, so it gets its own change set. Thanks for spotting.
>
> I know there would be careful thinking needed on how to design such intermediate layer and it would be a drastic change of the current API. I just wanted to make you aware of this idea. Maybe one could approach this in Jython 3 or so, or work out a minimal implementation of it, being open for more advanced use in the future.
>
>
> I think it would make sense to have a layer below BaseBytes that
> contained all those mechanisms that work without assuming a byte[]
> storage. This would help you implement PyBuffer in an object unable to
> export a byte[]. That wouldn't change the API and is likely harmless to
> efficiency. But more radical ideas, I agree, need more careful thought.
> (The present design has had a lot of thought.)
>
> Jeff
>
> ------------------------------------------------------------------------------
> Infragistics Professional
> Build stunning WinForms apps today!
> Reboot your WinForms applications with our WinForms controls.
> Build a bridge from your legacy apps to the future.http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
> _______________________________________________
> Jython-dev mailing lis...@li...https://lists.sourceforge.net/lists/listinfo/jython-dev
>
>
>