Re: CLISP support for a new portable FFI library

[James B. <ja...@ja...>]

"Hoehle, Joerg-Cyril" <Joe...@t-...> writes:

[ implementing my %MEM-REF operator: ]

> (ffi:foreign-value
>  (ffi::%offset (ffi:foreign-variable ptr type) offset type))

Thanks, this works.

> -- which could be made more efficient if need be.

Probably a good idea, this does seem to cons a bit much for a
low-level memory accessor.

> foreign-variable requiries clisp>2.33.2. If you don't have something
> that new, contact me (or search the clisp-list archives for keywords
> like cast% or foreign-variable or with-foreign-object).

I got this working in 2.33.2 by using the version of FOREIGN-VARIABLE
in your FFI patch available on SourceForge.  I wasn't able to build
the CVS version on OS X 10.4.1.

> I wrote <sort-of-foreign-value> to stress that your spec. is
> underspecified w.r.t. to the exact value. More precisely, it's not
> yet underspecified, because it only knows about some immediate types
> like int, single-float and (untyped) :pointer, so ffi:foreign-value
> fits the bill, but no struct or union types and pointers to such
> beasts, which start to make UFFI with CLISP a challenge.

Yes, structures and unions are implemented in terms of %MEM-REF.  For
example, there is a FOREIGN-SLOT-VALUE function used to access slots
in pointers to foreign structures:

(cffi:defcstruct timeval
  (tv-sec  :long)
  (tv-usec :long))

(cffi:with-foreign-object (tv timeval)
  (cffi:foreign-slot-value tv 'tv-usec))

Since the type information is available at compile time, there is
a series of compiler macros that expands FOREIGN-SLOT-VALUE to:

  (cffi-sys::%mem-ref tv :long 4)

In native-code implementations there is another compiler macro that
expands this into something like (sb-sys:sap-ref-32 tv 4), which turns
into inline assembly.  It would be nice if there was some fast C
primitive this could compiler macro expand into for CLISP.

> Whereas with CLISP, if CFFI code translates to lots of nested
> primitive %MEM-REF invocations and bytecode, it would be potentially
> slowed down stronger than implementations compiling to native code.

Agreed.  Hopefully my example above illustrates how I intend to use
%MEM-REF.

One higher-level thing that should probably be added to the CFFI-SYS
specification are functions for converting pointers to foreign strings
to and from Lisp strings, as I imagine there is a fast primitive to do
this in CLISP other than looping over characters and using (SETF AREF).

Also, I haven't looked into whether CLISP can do this, but the other
Lisps I support have a mechanism to pass a pointer to the data for
vectors of certain element types directly to foreign code (potentially
disabling the GC during such an operation, etc).  I would like to add
a high-level "shareable-vector" inteface to support this, which I hope
should handle many cases that are currently implemented with slow
code.

> OTOH, so far, my UFFI wrappers are oriented towards maximum
> performance.  Running the cl-sdl demos, CLISP and cmucl show the
> exact same fps rate, which shows that the FFI or even the Lisp is
> not the bottleneck in at least the cl-sdl demos.

Absolutely, performance is one of my goals as well.  For example, by
using my CFFI implementation of the UFFI interface, I was able to
reduce the consing when performing SQL queries in the CLSQL sqlite3
backend under SBCL by a factor of 10.

> a) better provide for a module name directly, or you'll get trouble
>    with Lispworks as well;
>
> b) CLISP can be made not to require a module name, but so far, we
>    (me at least) prefer it that way, as it is more portable and has
>    less restrictions! (in this particular point I'll change something
>    in the future).

I glanced at the ChangeLog for the CVS version and it seems like using
(:LIBRARY :DEFAULT) should do exactly what I want.  Once I get a CVS
CLISP building I'll try this out.

> I plan to export FFI::foreign-library.  AFAIK, foreign-library-close
> is already exported.

Cool.

> Why repeat functionality that's just built into many OS?  Edi Weitz
> is just proposing a patch to UFFI to let the OS search the library.

Er, that comment was sort of an indirect reference to what Edi is
going through with UFFI, which tends to implement the search using
FIND-FOREIGN-LIBRARY and wants absolute pathnames for libraries.  I
definately don't want to duplicated any functionality, just implement
it if the host Lisp doesn't.

> Look at (macroexpand-1'(ffi:def-call-out ...)) for insights.

Thanks, I was able to get this working.  Here's the macroexpansion I'm
using so far (I don't know if this use of LOAD-TIME-VALUE is safe, and
obviously I can't hardcode libSystem, I'm hoping to switch that to
:DEFAULT in the next version):

(FUNCALL
 (LOAD-TIME-VALUE
  (FFI::FOREIGN-LIBRARY-FUNCTION
   "sqrtf"
   (FFI::FOREIGN-LIBRARY "libSystem.dylib")
   NIL 
   #(FFI:C-FUNCTION SINGLE-FLOAT #(SINGLE-FLOAT 0) 1024)))
 16.0)

This seems to perform quite well.

Thanks for your help!

James