Re: CLISP support for a new portable FFI library

[Hoehle, Joerg-C. <Joe...@t-...>]

James Bielmann wrote:
[please also CC possible replies to me]

Contents:
o pointers to Lisp objects?
o block operations as a design rule
o compatibility of .fas files
o performance with FFI

>For example, by
>using my CFFI implementation of the UFFI interface, I was able to
>reduce the consing when performing SQL queries in the CLSQL sqlite3
>backend under SBCL by a factor of 10.
Ah, nice!

BTW, your %MEM-REF matches precisely the affi:MEM-READ I made up in 1994. See clisp/src/affi.d. It could read [un]signed-8/16/32 bit entities, pointers and also strings from address + offset. It could be extended to read single and double floats.


>Also, I haven"t looked into whether CLISP can do this, but the other
>Lisps I support have a mechanism to pass a pointer to the data for
>vectors of certain element types directly to foreign code (potentially
>disabling the GC during such an operation, etc).

IMHO you better forget about this immediately.
It's completely incompatible with
a) callbacks
b) threading
c) life-time considerations of the validity of the data pointed to (consider the moving GC) when you handle a pointer to a foreign entity: when will it dereference the data pointed to?
AFAIK Allegro has "pin this in memory until I tell you to release", but there's nothing like this in CLISP.
WITHOUT-INTERRUPTS-OR-GC doesn't cut it (a very limited design, just ok for hardware people). On the multitasking Amiga, use of Forbid()/Disable() was as frown upon as use of EVAL in CL :-)

The AFFI did pass pointers to Lisp arrays, but there where a) no callbacks (most OS callbacks would have been run in a b) different process anyway, so that would not work from clisp). c) Some calls were synchronous (e.g. Read()), but not all.  E.g. SetWindowTitle(title) expects the buffer holding the title to hold until another title (pointer) is set -- not a place for a pointer to a Lisp string.
And it was before #+unicode, so the unique encoding was the OS'encoding: ISO-8859-1. And I flagged it as dangerous: CLISP strings are not \0-terminated.
Nowadays, CLISP maintains strings in three different formats dynamically: 8/16 or 32 bits wide, and foreign code is not prepared to deal with that.

If you really want to access CLISP internals, you write an external module. But not an UFFI/CFFI/AFFI.


>Since the type information is available at compile time, there is
>a series of compiler macros that expands FOREIGN-SLOT-VALUE to:
>(cffi-sys::%mem-ref tv :long 4)

Well, in CLISP, there's another design goal you may wish to consider, which I appreciate: portability of .fas files. I very much appreciate it when I can take my application's .fas files and use that with any clisp of the same version.
That means that alignment and size may not be known until load-time.

Luckily, code generated for
(cffi-sys::%mem-ref tv :long (load-time-value (offset-after '(int + (struct foo) + char (array char 2)))))
is just as efficient as the directly inlined constant. It just takes a little longer to load (and some thinking on the availability of type and struct definitions at load-time).

LOAD-TIME-VALUE is your friend once again.

LOLAD-TIME-VALUE is not so nice in the interpreter, because there it's basically IDENTITY and thus re-evaluated each time in a loop. In the interpreter, one would wish for Scheme's (FORCE (LOAD-TIME-VALUE (DELAY ...))) to be built into load-time-value instead of the current behaviour.


>to and from Lisp strings, as I imagine there is a fast primitive to do
>this in CLISP other than looping over characters and using (SETF AREF).

One design rule of mine is: provide for "block" operations. I.e. if all you have is READ-CHAR (or %MEM-REF), your design is broken (although fine from the academic POV). I could give many examples where it applies, from practical programming to networking and hard disk controllers and even GUIs.

I even went so far so as to consider some month ago to import a variant of affi:mem-write-vector and mem-read-vector into ffi:, i.e. omit the "one element" accessors and provide only the array based ones. Of course, the context there was that ffi: already has enough element-wise accessors, but needs enhancements in the domain of arrays of variable size.

Providing block operations in addition to element-based ones lowers the part of redundant "administrativia" (e.g. setup, function calls) vs. the core operation (e.g. get bytes copied).

BTW, the AFFI:MEM-READ is able to copy into CLISP's specialized (unsigned-byte 8/16/32) vectors -- a tight, efficient loop. There's no provision for signed entities, nor arrays of floats, for which there's no specialized representation in CLISP.


>[foreign-variable] Probably a good idea, this does seem to cons
>a bit much for a low-level memory accessor.
The FFI FOREIGN-VARIABLE objects, although at the low-level of the FFI, cons a lot, since all this type information stored in slots is kept in objects, while your proposal is to compile away all this cruft.

A performance killer in the CLISP FFI is IMHO:
(dotimes
  (setf (element (deref x) i) foo))
This causes 2*N foreign-variable objects to be allocated, filled, and thrown away a little later, many many type checks and other extra work.
Compare this to native code compilers, where the loop could be like hand-written tight assembly code. 2 order ot magnitude of difference in speed! (1x bytecode interpreter, 1x FFI type interpreter).

In CLISP, it pays off to do constant expression folding:
(with-c-var (p (deref x))
  (dotimes # (setf (element p i) foo)))
-> Only 1*N throw away objects.

Also, what costs time is the explicit representation of all things as pointers, i.e. (slot foo 'x) is
(foreign-value (%slot foo 'x))
and conses an extra foreign-variable object for each slot access.
To compare with mem-read, which may not cons at all.

Sadly, UFFI code is likely to do the above, making it slower than code written to the CLISP FFI, which would call FOREIGN-VALUE on the whole array, letting CLISP do the copying fast (remember above about block design?):
(deref x) [as place] -> Lisp array(!)


Finally, you may want to know that in CLISP, foreign-variable indirect to foreign-address, which itself indirects to foreign-pointer. I.e. (SLOT foo 'x) causes cration of one foreign-variable and one foreign-address object, and each access goes via three indirections(!)

Yet using #<FOREIGN-ADDRESS> instead of #<FOREIGN-POINTER> (as the AFFI did) has some advantages. In fact, it's the Amiga that influenced the FFI to represent addresses as a base pointer plus an offset. And that still is at the core of the FFI today, although very little use is made of that. It is supposed to be a good match for (D)COM, but I never investigated that.
On the Amiga, it was used for dependency tracking and sharing validity of pointers (see #'(SETF VALIDP) in the FFI)

In summary, you can see that there are plenty of ways to improve the efficiency of the CLISP FFI (possibly at the cost of type safety). Hopefully the .fas file compatibility issue will matter to you.

Regards,
	Jorg Hohle.