Re: CLISP support for a new portable FFI library

[Hoehle, Joerg-C. <Joe...@t-...>]

James Bielmann wrote:

>I wasn't really specific enough here.  The use cases I'm concerned
>with are when you have a vector of (UNSIGNED-BYTE 8)'s and you want to
>pass it to some C function that's either going to fill it, or consume
>it (possibly copying it elsewhere).

>My current proposed interface consists of MAKE-SHAREABLE-BYTE-VECTOR
>and WITH-POINTER-TO-VECTOR-DATA, which I describe in CVS at:
>http://common-lisp.net/cgi-bin/viewcvs.cgi/*checkout*/cffi/doc/
>shareable-vectors.txt?rev=HEAD&cvsroot=cffi

I see the following "portability challenge": there's no separation between "return is needed" vs. return thrown-away. Yet it makes a heck of a difference.
"read" is very different from "glTexImage2D" in that aspect.

Consider for example a library whose author considers with-pointer-to-data a very cool efficiency thingy ("get the pointer to Lisp (UNSIGNED-BYTE 8) vector, that's it!"), and s/he uses it everywhere.
Implementation like clisp would be hosed, because in many occasions, the write-back could be a waste of resources.

Also, there's a problem with that that when people see that this macros is useable with vectors or other types (not only :uin8) in implementation XYZ, they'll start to use that for other types too, and forget that other implementations don't support that.

Again, I strongly advocate declarative interfaces (e.g. :in, :in-out etc.) so the CFFI can automatically derive the best code: copy-back can be eliminated via interface descriptions that the programmer provides (e.g. not for :in).


>By using WITH-POINTER-TO-VECTOR-DATA the programmer is asserting that
>foreign code won't hang on to the pointer beyond that dynamic contour
>or all kinds of undefined behavior will result.
What does the programmer know about how callbacks may interact with threads in a given CL implementation?

What if simple tests (no other threads running) work for the library author, while complex environments (several threads running other CL packages or applications) cause unexpected failure because the moving GC comes in?

At least, you should mention that if implementating a constant pointer requires without-gc-interrupts, then it's better not to use that hack in that implementation of CL!


>> One design rule of mine is: provide for "block" operations.
>I went to specify versions of MEM-{READ/WRITE}-VECTOR for CFFI and
>essentially came up with the same interface as you have in AFFI, so if
:-) well, thanks :-)

>you did do this port I think that'd be great for CFFI.

Actually, when I thought about copying it for FFI, I thought about some changes, e.g. use :start and :end keywords for the Lisp vector. That could have good uses (e.g. cl-pdf:compress). And I never planned on using :COUNT, but rather limit the number via :start and :end.
Having both looks redundant at first glance.

:start/end feels Lispy, :count C-ish.
But I haven't analyzed whether :count would be more useful in the context of an FFI -- i.e. count might be the value that other functions like compress() return, so translating that to :start and :end means extra code, causing :count to be the more useful interface than :start/end.

>Here's the proposed interface:
>http://common-lisp.net/cgi-bin/viewcvs.cgi/*checkout*/cffi/doc/
mem-vector.txt?rev=HEAD&cvsroot=cffi
[haven't reviwed that in depth]

The "equivalent" code is bogus in the "VECTOR is not large enough" case.

Another low-level efficiency issue: I believe &optional to be generally more efficient than :key, but it probably depends a lot on the implementation (and the low-level entry calls, block compilation, compiler settings etc.)

So why not
mem-read-vector vector ptr type &optional count (offset 0)
; cout not set means limit given by Lisp vector.


Certainly, no key is fastest in all implementations (e.g. cmucl produces much fewer entry points), so you might consider that as the low-level macros/functions.
Whether no &optional is fastest, I don't know. In CLISP, some entity must push the #<unbound> object on the stack for the callee to detect it. As a result:

# (disassemble(lambda(s)(read s)))
Disassembly of function :LAMBDA
1     (PUSH-UNBOUND 3)
Which means less bytecodes than 3 individual push eof-error-p, push eof-value, push recursive-p bytecodes.
Thus &optional with many optionals seems faster than always providing the defaults.

Anyway, we're probably not discussing the key performance hot spots of any FFI-using application here, so that doesn't matter much.

Actually, thinking about that again it would be nice if you did not strictly mandate a lowest-level interface, so each implementation could do what is fastest. This remembers me of Eric Marsden's pg.lisp which suffers from the fact that some implementations can better read double-float from foreign pointers, others from streams and others from unsigned-byte arrays. pg.lisp forces one choice, making others slower than if directly written using the implementation's useful extensions.


Function: MEM-WRITE-C-CHAR is IMHO badly designed w.r.t. multi-byte etc. Having no length limitation (in bytes) is like a guarantee for a buffer overflow some time, esp. with multibyte.

>Function: mem-read-vector vector ptr type count &key (offset 0)
You do not specify whether it's acceptable to copy e.g. type :uint16 into a vector specialized on (unsigned-byte 32).
That's the problem with having both type and array-element-type (implicit via vector) arguments.


>Yes, ideally I could use MEM-READ-VECTOR with a count of 1 on a
>foreign pointer to avoid consing up this type information.
Wasn't there the (make-array ()) 0-rank thread in cll month ago?
(make-array () :element-type '(unsigned-byte 8))
(setf (aref *) 123)
; not sure my AFFI groks such beasts
;-)


I'm surprised you don't mention Lisp vector creating functions, out of a pointer. Is that available at a higher-level?


>I admit to punting on Unicode so far---
CLISP's FFI also lacks some foreign-pointer/to/from-string primitives (not yet available as UFFI macro wrappers).

Actually, IMHO CLISP needs to (define &) export some more multibyte functions, e.g. foreign-string-length is certainly needed as soon as foreign-porinter/to/from-string would be available.

I'm happy to see that you also have def-c-struct etc. macros ready, i.e. that your work seems quite advanced. What UFFI-using libraries did you test?

I'll look into adding some variant of affi:mem-read/write into clisp's FFI (please don't hold your breath, it's been on my TODO lists for a long time already).

Regards,
	Jorg Hohle.