[clisp-list] FFI and variable length arrays

[John S. <jwe...@co...>]

Clisp FFI question: variable length arrays returned from C function that are 
not null terminated


Problem:

I'm trying to use CLisp FFI with a C function that returns a pointer
to a variant length array of strings (i.e. char *) but the array is not null 
terminated.

I've distilled the essense of my problem down to the example below calling 
out to "foo".

FYI the real problem involves interfacing to MySQL's mysql_fetch_row which 
returns
a non terminated array of column values. And SQL NULL values unfortunately 
are
returned a NULL pointers to boot further complicating the issue.  The number 
of
columns are returned by a separate function: mysql_num_fields and 
unfortunately
there is no other function in the API such as mysql_fetch_field_value() 
which would
allow me to work around the problem.  I've written a simple DBI/DBD wrapper 
including
ODBC (which works) and MySQL (which almost works) and will be posting it as 
open
source as soon as it is completed. I'm also working on a wrapper for Fltk 
for CLisp.

Thanks to Joerg for his previous answer. I've tried implementing his 
suggestions
but I failed.  Below are files giving a minimal example (highlighting my
misunderstanding).  Hopefully they are small enough that 
comments/corrections
can be inserted directly at the locations of my bugs and require minimum 
effort
to answer this email.  Thanks for any help!!

(BTW I'm using CLisp 2.33.1 on win32 from prebuilt binaries.  (I don't
know how to use CVS yet and failed in my attempt to build under MSYS
that I just installed using the new 2.33.2 sources - still learning this
environment too.  So I don't have FOREIGN-VARIABLE constructor yet.)

Solution: (attempted but failed)

---------------------------------------------------------------------------------------------------
// foo.c

#include <stdio.h>

// define for Microsoft Visual C on win32
#define DLLEXPORT _declspec(dllexport)

// Otherwise
// #define DLLEXPORT


char * cols[] = { "one", "two", "three", "four" };

// Real foo will return variant length array that is
// not null terminated.

DLLEXPORT char ** foo()
{
  printf("\nC address of string array: %x",cols);
  return cols;
}

---------------------------------------------------------------------------------------------------------------

;; Example of accessing arrays that are not null terminated
;; foo.lisp
;; see foo.c

(FFI:DEF-CALL-OUT      foo
  (:library            "foo.dll")
  (:language           :stdc)
  (:name               "foo")
  (:arguments)
  (:return-type        (ffi:c-ptr (ffi:c-array ffi:c-string 3))))

;; The real foo will return varying sized arrays that
;; are not null terminated so ffi:c-array-ptr can not be used.
;; This call out spec simply tests a fixed size array with a
;; specified length less than or equal to the actually array returned.

(setf s (foo))

(format t "~%Two: ~A" (aref s 1))

(format t "~%'one' 'two' 'three': ~A" s)

(setf col-count 3) ;; varies on the real foo.

;; this version generates the error indicated below

(FFI:DEF-CALL-OUT      bar
  (:library            "foo.dll")
  (:language           :stdc)
  (:name               "foo")
  (:arguments)
  (:return-type        ffi:c-pointer))

(setf s2 (bar))

(format t "~%Foreign address: ~A" s2)


;; convert foreign address to c-place for FFI primitives

(ffi:with-c-var (cols `ffi:c-pointer s2)

  (ffi:offset cols 1 `(ffi:c-array ffi:c-string ,col-count))

;; Above line generates this error:
;; *** - FFI::%OFFSET: foreign variable #<FOREIGN-VARIABLE
;; "EXEC-ON-STACK" #x00..> does not have the required alignment

)


;; Alternate approach - also generates error indicated below.

(FFI:DEF-CALL-OUT      foobar
  (:library            "foo.dll")
  (:language           :stdc)
  (:name               "foo")
  (:arguments)
  (:return-type        (ffi:c-ptr (ffi:c-array ffi:c-string 1))))

(setf s3 (foobar))

(format t "~%Two: ~A" s3)

(ffi:with-c-place (cols s3)

  (ffi:offset cols 1 `(ffi:c-array ffi:c-string ,col-count))

;; Above line generates this error:
;; *** - FFI::%OFFSET: argument is not a foreign variable: #("one")

)

I don't have a conceptual model in my mind on what is really going
on under the covers and it is still not clear to me what is meant
by foreign.  My understand so far is as follows:

I figure the FFI spec such as a call out function is kind of like IDL
which the system (or VM) is referencing when a call out is made to
determine how it will throw values across the wall surrounding GC
managed memory out to where the C code can get to it without fear
of a GC attack.  And likewise on values returned from C (other than 
c-pointer)
the VM has to call into play allocation, copying and perform a possible 
format
transformation to bring this returning data back within the GC memory 
compound
making it a bona fide Lisp object.

What is confusing to me is the naming conventions of foreign and c-place. It 
seems
it means several things depending on where it is in bound, outbound or a 
settable
place. I haven't thought this all through yet but I'm tempted to try to 
document
this more fully for my own edification so that I truly understand what's 
going
on so I'm not unwittingly making mistakes.  However I realize the FFI is 
designed
so painstakingly so that you can't do unsafe things - which is perhaps why 
I'm
having such a hard time with getting at foo's returned value above.

Thanks for any help.  Looking forward to having a working DBI/DBD and 
evently
a Fltk bindings.

John
p.s. Thank you for writing CLisp!