Re: CLISP support for a new portable FFI library

[James B. <ja...@ja...>]

"Hoehle, Joerg-Cyril" <Joe...@t-...> writes:
> James Bielmann wrote:
>
>>Also, I haven"t looked into whether CLISP can do this, but the other
>>Lisps I support have a mechanism to pass a pointer to the data for
>>vectors of certain element types directly to foreign code (potentially
>>disabling the GC during such an operation, etc).
>
> IMHO you better forget about this immediately.
> It's completely incompatible with
> a) callbacks
> b) threading
> c) life-time considerations of the validity of the data pointed to
>    (consider the moving GC) when you handle a pointer to a foreign
>    entity: when will it dereference the data pointed to?

I wasn't really specific enough here.  The use cases I'm concerned
with are when you have a vector of (UNSIGNED-BYTE 8)'s and you want to
pass it to some C function that's either going to fill it, or consume
it (possibly copying it elsewhere).

Examples would be calling "read" fill a vector with data read from a
socket or "glTexImage2D" to load image data from a Lisp vector into
OpenGL.

My current proposed interface consists of MAKE-SHAREABLE-BYTE-VECTOR
and WITH-POINTER-TO-VECTOR-DATA, which I describe in CVS at:

   http://common-lisp.net/cgi-bin/viewcvs.cgi/*checkout*/cffi/doc/shareable-vectors.txt?rev=HEAD&cvsroot=cffi

By using WITH-POINTER-TO-VECTOR-DATA the programmer is asserting that
foreign code won't hang on to the pointer beyond that dynamic contour
or all kinds of undefined behavior will result.

I've tried to design this such that a CFFI-SYS implementation can
implement shareable byte vectors by copying in/out from a locally
allocated buffer if it can't get a pointer directly into the vector
(which is what you'd end up doing anyway without this interface).

Presumably CLISP can use the minimal (copying) version.

> And it was before #+unicode, so the unique encoding was the
> OS'encoding: ISO-8859-1. And I flagged it as dangerous: CLISP
> strings are not \0-terminated.  Nowadays, CLISP maintains strings in
> three different formats dynamically: 8/16 or 32 bits wide, and
> foreign code is not prepared to deal with that.

Absolutely, Lisp strings are not shareable vectors in CFFI.

>>Since the type information is available at compile time, there is
>>a series of compiler macros that expands FOREIGN-SLOT-VALUE to:
>>(cffi-sys::%mem-ref tv :long 4)
> Well, in CLISP, there's another design goal you may wish to
> consider, which I appreciate: portability of .fas files. I very much
> appreciate it when I can take my application's .fas files and use
> that with any clisp of the same version.  That means that alignment
> and size may not be known until load-time.

Yes, this definately is a nice feature of CLISP.

> Luckily, code generated for
> (cffi-sys::%mem-ref tv :long (load-time-value ...))
> is just as efficient as the directly inlined constant. It just takes
> a little longer to load (and some thinking on the availability of
> type and struct definitions at load-time).

I think this should work fine, all the type information available at
compile-time should also be available at load-time.  I'll need to cook
up some tests that move .fas files across platforms to try this all out.

> LOAD-TIME-VALUE is not so nice in the interpreter, because there
> it's basically IDENTITY and thus re-evaluated each time in a
> loop. In the interpreter, one would wish for Scheme's (FORCE
> (LOAD-TIME-VALUE (DELAY ...))) to be built into load-time-value
> instead of the current behaviour.

Hopefully I can get away with telling folks to compile their code
then! :-)

>>to and from Lisp strings, as I imagine there is a fast primitive to do
>>this in CLISP other than looping over characters and using (SETF AREF).
>
> One design rule of mine is: provide for "block" operations. I.e. if
> all you have is READ-CHAR (or %MEM-REF), your design is broken
> (although fine from the academic POV). I could give many examples
> where it applies, from practical programming to networking and hard
> disk controllers and even GUIs.
>
> I even went so far so as to consider some month ago to import a
> variant of affi:mem-write-vector and mem-read-vector into ffi:,
> i.e. omit the "one element" accessors and provide only the array
> based ones. Of course, the context there was that ffi: already has
> enough element-wise accessors, but needs enhancements in the domain
> of arrays of variable size.

I went to specify versions of MEM-{READ/WRITE}-VECTOR for CFFI and
essentially came up with the same interface as you have in AFFI, so if
you did do this port I think that'd be great for CFFI.

Here's the proposed interface:

http://common-lisp.net/cgi-bin/viewcvs.cgi/*checkout*/cffi/doc/mem-vector.txt?rev=HEAD&cvsroot=cffi

I admit to punting on Unicode so far---at least with MEM-*-C-STRING
defined separate from MEM-*-VECTOR there is a place to hang whatever
sort of :EXTERNAL-FORMAT arguments are necessary to do Unicode right.

> The FFI FOREIGN-VARIABLE objects, although at the low-level of the
> FFI, cons a lot, since all this type information stored in slots is
> kept in objects, while your proposal is to compile away all this
> cruft.

Yes, ideally I could use MEM-READ-VECTOR with a count of 1 on a
foreign pointer to avoid consing up this type information.

> In summary, you can see that there are plenty of ways to improve the
> efficiency of the CLISP FFI (possibly at the cost of type
> safety). Hopefully the .fas file compatibility issue will matter to
> you.

Absolutely, thanks for all your feedback!

James