[clisp-list] FFI: variable size values (was FFI crash: what am I missing?!)

[Hoehle, Joerg-C. <Joe...@t-...>]

Hi,

Sam wrote:
>> >what is the correct idiom for the C "unknown type array"?
>> >E.g., suppose
>> >        int* foo();
>> >will return the array of integers of size N, where N is the return
>> >value of another function
>> >        int foo_size();
>> >how do I express foo() and foo_size() in CLISP FFI?

>> >Suppose
>> >        int foo (int *size, int**arr);
>> >returns status and puts the length of the array into `size' and the
>> >pointer of the array into `arr'.
>> >what is the correct CLISP FFI idiom?
See answers below

>what if, as in the first example, the size is returned by a _different_
>function?!
That's why I said "the latter case" fits my (unwritten) macro. The former does not fit the one declaration per function" model, since the "transaction" (I name is so) is distributed across two calls.

>> Explicit management using WITH-FOREIGN-OBJECT and friends is
>> involved.

>any cast requires equal memory size.
I wrote in the past on the OFFSET solution.
It looks like you have forgot about my article "FFI howto: variable length arrays (long)" from 5th of March 2003.
http://sourceforge.net/mailarchive/forum.php?thread_id=1784502&forum_id=6767
http://sourceforge.net/mailarchive/forum.php?thread_id=1788500&forum_id=6767


>is a C wrapper the only solution?
Not at all. Sometimes I recommend C wrappers to create an interface that better suits Lisp, but it is not strictly necessary. You can use the FFI to call any function and retrieve its results, writing Lisp code (be it less efficient because of run-time parsing (FFI:parse-c-type) and conversions (ffi:cast)).
But see my below comment on the regexp case.

>> A longer answer may follow tomorrow, I need to go home now.
>biting my nails off :-)
I hope you can still type in the following:


			The first foo
You didn't say whether foo_size() has to be called before or after foo(). I'll call it afterwards, for similarity with the regexp module, where the number of the group positions can be read off a slot of the structure.

BTW, the current CLISP regexp module arbitrarily limits the return to 10 group positions ("num-matches"), because it does not consider the re_nsub slot out off the regex_t structure.

This is a case where I'd recommend a little C wrapper code to make interfacing easier. I very much prefer this over duplicating the regex_t structure in Lisp via (FFI:C-STRUCT <many slots>). I don't want to have to maintain this by hand, whereas the C wrapper would always stay in sync with the C regex.h header by mere recompilation.

(def-call-out %foo1 (:name "foo")
  (:return-type c-pointer) (:language :stdc))
(defun foo1 ()
  (let ((ptr (%foo1)))
    (when ptr ;; c-pointer type will return NIL for NULL in the near future
      (with-c-var (array 'c-pointer ptr)
        (cast array `(c-ptr (c-array sint ,(foo-size))))))))


			The second foo
I take it that foo2 allocates the buffer it returns via arr . Who must free it?
Also you don't say whether it's :in-out or :out, so I take :out only.
However it's common for the buffer to be supplied by the caller, with a maximum size provided by the size argument as an :in-out parameter.
The C declaration you supply doesn't tell the difference, and the below code covers the other case.

Here's what the macro call might like
(def-call-var-out foo2 (:name "foo")
  (:arguments (size (c-ptr uint) :out)
              (arr (c-ptr (c-ptr (c-array sint size))) :out [:malloc-free]))
  (:guard (zerop return))
  (:return-type int) (:language :stdc))

BTW, I recommend using unsigned types even though C people usually just use "int" (new-wave C people say "size_t"). E.g. size obviously should be unsigned.

Here's what you do without the charming macro:
(def-call-out %foo2 (:name "foo")
  (:arguments (size (c-ptr uint) :out)
              (arr (c-ptr c-pointer) :out))
  (:return-type int) (:language :stdc))

(defun foo2 ()
  (multiple-value-bind (status size array-ptr)
      (%foo2)
    (if (zerop status)
      (with-c-var (arr 'c-pointer array-ptr)
        (cast arr `(c-ptr (c-array sint ,size))))
      ;; the macro would return (values status [size array])
      (error "Call failed with status ~D" status))))


This costs a PARSE-C-TYPE for every call for both foo1 and foo2.

My MEM-READ proposal could avoid that in some cases, depending on the type (so far only built-in-arrays of CLISP, i.e. uint8/16/32, but not sint8/16/32 are supported).
(defun foo2 ()
  (multiple-value-bind (status size array-ptr)
      (%foo2)
    (if (zerop status)
        (let ((vec (make-array size :element-type '(unsigned-byte 32))))
          (mem-read array-ptr vec))
      (error "Call failed with status ~D" status))))
(defun foo1 ()
  (let ((ptr (%foo1)))
    (when ptr ;; c-pointer type will return NIL for NULL in the near future
      (let ((vec (make-array (foo-size) :element-type '(unsigned-byte 32))))
        (mem-read ptr vec)))))

My FOREIGN-VARIABLE constructor (design not yet finished) would allow to avoid WITH-C-VAR, but would still require PARSE-C-TYPE at run-time, allowing code similar to:
(defun foo1 ()
  (let ((ptr (%foo1)))
    (when ptr ;; c-pointer type will return NIL for NULL in the near future
      (foreign-value
       (foreign-address-variable ptr `(C-ARRAY SINT ,(foo-size)))))))
or maybe I'll require an explicit call
       (foreign-address-variable ptr
        (parse-c-type `(C-ARRAY SINT ,(foo-size))))
It's a constructor after all, not a macro-level keep-typing-short sort of thing.

Regards,
	Jorg Hohle.