Re: [Jython-dev] Jython buffer protocol

[Jeff A. <ja...@fa...>]

I'm giving serious consideration to idea 2, that is, the storage 
implementation is j.n.ByteBuffer, always, and *may* wrap a byte[] 
object. I'd need to try this out to ensure there is no fatal flaw.

*Jim:* this is a breaking change to the API. Do we need to be more 
careful of possible users? I suspect we are only breaking our own work 
here: how about you?

We would be saying in this that the Jython PyBuffer is allowed to be 
less like the CPython one than I've been aiming for. This consistency 
may be less important than Stefan's use case. The CPython protocol 
promises efficient access to the storage of an object via a pointer, and 
we would be saying "only as efficient as a j.n.ByteBuffer" ... although 
it may turn out there's a backing array. j.n.ByteBuffer does not replace 
PyBuffer, because it cannot describe strided access or the get-release 
behaviour.

I think this leads to an API in which what I've tried to do with 
PyBuffer.Pointer we now do by handing out ByteBuffer slices. So Pointer 
goes away. In that case getBuf() and getNIOByteBuffer() are probably the 
same thing. I do not think it is safe to hand out the actual storage: it 
is almost unavoidable clients would manipulate the internal state 
(position, limit), surprising each other and the PyBuffer implementation 
if it relies on them, as I think it should.

Concerning the pointer to object member in CPython Py_Buffer, it seems 
to be a 3.x feature, which may be why I haven't replicated it. It seems 
easy to add. (I'd be rewriting all the constructors anyway.) In CPython 
it's null when there's a buffer but no object.

Jeff Allen

On 24/04/2016 15:36, Stefan Richthofer wrote:
> Jeff,
>
> good to hear that you can help with this stuff and also that your answer implies you don't have concerns with the new feature itself. Thinking it through again, I think the following way would be cleanest to add this functionality:
>
> Add a ByteBuffer-type storage, either exclusively or in addition to byte[] storage.
>
>
>
> 1) Version with additional field java.nio.ByteBuffer bufferStorage:
>
> Case byte[]-backed PyBuffer:
> (buffer storage must be view on storage, i.e. backed by it and must always point to first element)
>
> storage is byte[]
> bufferStorage is ByteBuffer.wrap(storage)
>
> getNIOByteBuffer() can use bufferStorage and needn't call ByteBuffer.wrap every time again.
>
>
> Case direct ByteBuffer (likely not having backing array):
>
> storage is null or if the JVM happens to be capable of providing direct ByteBuffer with byte[] backend: bufferStorage.array()
>
> bufferStorage is ByteBuffer.allocateDirect(capacity)
>
> Methods that used to access elements of storage directly are enriched by a fallback for case storage == null. The fallback would directly operate on bufferStorage.
>
>
>
>
> 2) Version with exclusive Buffer-storage:
>
> storage type is java.nioByteBuffer instead of byte[]
>
>
> Case byte[]-backed PyBuffer:
>
> storage is ByteBuffer.allocate(capacity) (i.e. non-Direct, so buffer will have backing array!)
>
> getNIOByteBuffer() can use storage and needn't call ByteBuffer.wrap.
>
> Methods that used to access elements of storage directly now do this on storage.array() rather than on storage itself (should be doable by a simple search/replace refactoring more or less).
>
>
> Case direct ByteBuffer (likely not having backing array):
>
> bufferStorage is ByteBuffer.allocateDirect(capacity)
>
> Methods that used to access elements of storage directly are enriched by a fallback for case storage.hasArray() == false. The fallback would directly operate on storage's ByteBuffer methods.
>
>
> I can do the work of writing the fallbacks or help with it up to your discretion.
>
>
> Then another thing: I noticed CPython's PyBuffer-pendant contains a reference to the PyObject that exported it, so you can always find the origin of a given PyBuffer. I don't see how this would be feasible with Jython's current PyBuffer implementation. So from JyNI perspective I can store (as a mapping) the exporter in case it is known for some reason, e.g. because PyBuffer was converted from a native CPython-like variant.
> However there could be situations where the buffer comes from Jython and the origin would be unknown. In that case I would (currently) just provide a NULL-value or PyNone for this field and hope to get away with it for the important extensions. Maybe we could attach a PyBuffer's origin in Jython too...? (e.g. as a JyAttribute only if some global flag is set, which JyNI would then set on load).
>
> Best
>
> Stefan
>
>
>
>> Gesendet: Samstag, 23. April 2016 um 20:14 Uhr
>> Von: "Jeff Allen" <ja...@fa...>
>> An: "Stefan Richthofer" <Ste...@gm...>
>> Cc: jim...@py...
>> Betreff: Re: [Jython-dev] Jython buffer protocol
>>
>> Hi Stefan.
>>
>> Refreshing my memory about how these classes work, I can see that I took
>> at face value the CPython view that the purpose of the buffer interface
>> is to give clients access to the underlying array of bytes, so
>> abstraction of the storage always gave way to what I thought would be
>> efficient. (Abstraction of the unit to be something other than byte is
>> sketched but clarity and a use case eluded me.)
>>
>> I always feel I've failed if I have to cast. My instinct is for option a.
>>
>> But I think you would not create a "Direct" parallel to BaseBuffer,
>> since it contains a lot of helper methods independent of the storage
>> implementation. Rather, factor it into two layers, the first being
>> either BaseBuffer or AbstractBuffer (depending on what causes least
>> pain) and the next layer being two base classes, one the revised
>> BaseBuffer containing:
>>       protected byte[] storage;
>> and the other containing:
>>       protected ByteBuffer storage;
>> And in each you migrate case whatever it seems natural should come along
>> with these declarations.
>>
>> I've been meaning to get back to Jython: I could do this groundwork if
>> that would not be confusing.
>>
>> Jeff
>>
>> Jeff Allen
>>
>> On 22/04/2016 21:50, Stefan Richthofer wrote:
>>> Hello Jeff,
>>>
>>> I'm warming up this old thread, because I am about to start actual work on JyNI's support
>>> for buffer-protocol / the PyBuffer builtin type.
>>> I'd like to point you to my recent pull request https://github.com/jythontools/jython/pull/39.
>>> It's a preliminary step for adding support for direct java.nio.ByteBuffers. After establishing this flag
>>> I am going to add some actual support for it. I see basically two ways to go for this
>>>
>>> a) Create a parallel class hierarchy to BaseBuffer et al, backed by direct ByteBuffers. E.g.
>>> call everything with "Direct": DirectBaseBuffer, DirectSimpleBuffer etc.
>>> Then let BufferProtocol implementers check for the flag and use Direct counterpart of the
>>> usually used Buffer-Class accordingly.
>>>
>>> or
>>>
>>> b) Modify existing BaseBuffer such that storage is Object rather than byte[]. Then according to
>>> flags it will be byte[] or ByteBuffer. This variant will involve more explicit type casting than
>>> a), but would involve fewer new classes however.
>>>
>>> What is your opinion about this?
>>>
>>> Best
>>>
>>> Stefan
>>>
>>>
>>>> Gesendet: Donnerstag, 07. August 2014 um 21:49 Uhr
>>>> Von: "Jeff Allen" <ja...@fa...>
>>>> An: "Stefan Richthofer" <Ste...@gm...>
>>>> Cc: "Jython Developers" <jyt...@li...>
>>>> Betreff: Re: [Jython-dev] Jython buffer protocol
>>>>
>>>>
>>>> Jeff Allen
>>>>
>>>> On 06/08/2014 02:28, Stefan Richthofer wrote:
>>>>> Jeff,
>>>>>
>>>>> I just quickly scanned the changes and everything looks fine as far as I see. PyByteArray and BaseBytes may need adjustments too (on this occasion remember to add resizeCheck() in irepeat).
>>>> Thanks for looking that over. I was rather asking whether the use case
>>>> was served adequately by the addition of PyBUF.AS_ARRAY, hasArray() and
>>>> getNIOByteBuffer(), since you obviously have a clear idea of it.
>>>>
>>>> I took a second/third look but didn't find anything to change in
>>>> BaseBytes and PyByteArray: when arguments objects have the Buffer API
>>>> they are accessed through the abstract API (not as byte[]). PyByteArray
>>>> obviously can support AS_ARRAY, and that seems to be covered in
>>>> SimpleBuffer.
>>>>
>>>> I fixed the irepeat bug in the previous change set: it's a distinct
>>>> issue, so it gets its own change set. Thanks for spotting.
>>>>> I know there would be careful thinking needed on how to design such intermediate layer and it would be a drastic change of the current API. I just wanted to make you aware of this idea. Maybe one could approach this in Jython 3 or so, or work out a minimal implementation of it, being open for more advanced use in the future.
>>>>>
>>>> I think it would make sense to have a layer below BaseBytes that
>>>> contained all those mechanisms that work without assuming a byte[]
>>>> storage. This would help you implement PyBuffer in an object unable to
>>>> export a byte[]. That wouldn't change the API and is likely harmless to
>>>> efficiency. But more radical ideas, I agree, need more careful thought.
>>>> (The present design has had a lot of thought.)
>>>>
>>>> Jeff
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Infragistics Professional
>>>> Build stunning WinForms apps today!
>>>> Reboot your WinForms applications with our WinForms controls.
>>>> Build a bridge from your legacy apps to the future.
>>>> http://pubads.g.doubleclick.net/gampad/clk?id=153845071&iu=/4140/ostg.clktrk
>>>> _______________________________________________
>>>> Jython-dev mailing list
>>>> Jyt...@li...
>>>> https://lists.sourceforge.net/lists/listinfo/jython-dev
>>>>
>>