Embeddable Common-Lisp

On Sun, 02 May 2010 22:04:50 -0500
Seth Burleigh <seth@...> wrote:

> I was wondering if it is currently possible to embed toplevel c/c++ code
> in a lisp file (structure definitions, but most importantly c/c++
> functions/methods). Just some ideas, i was wondering how 'difficult' the
> below would be, if it is worth it, etc.
[...]

Do you mean something more than the current FFI facilities (including
FFI:C-INLINE and FFI:CLINES)?  Because inline code can already be typed
litterally in lisp code, and lisp code can also generate dynamic C code
using those.

An example which heavily uses C-INLINE and CLINES for both litteral
inline C and dynamic C code generation can be found at:

http://cvs.pulsar-zone.net/cgi-bin/cvsweb.cgi/mmondor/mmsoftware/cl/test/ecl-unix.lisp?rev=1.1;content-type=text%2Fplain

Note that this code needs cleaning up and polishing, however.
Some fill functions are dynamically generated to serialize between C
and CL structures using CLINES.  Also, C-INLINE is used by the
defsyscall macros and for other litteral code.  C-INLINE is also used
by the code which generates CL accessor functions for C structures
(where this interface is prefered to fully serializing structures).
There is also litteral code inserted to deal with multiple BSD socket
address families and hash/compare addresses, as well as to deal with
unicode and with kevent arrays...

It's possible that you also meant higher level access than
CLINES/C-INLINE though, sorry if I'm stating the obvious in that case.
-- 
Matt

Currently, we can only input inline c code (i think) into either
(A) lisp functions
(B) Header file

But we cant inline arbitrary c code into the toplevel, like a c
function, a c++ class method, etc.

On Mon, May 3, 2010 at 1:03 PM, Seth Burleigh <seth@...> wrote:

> Currently, we can only input inline c code (i think) into either
> (A) lisp functions
> (B) Header file
>
> But we cant inline arbitrary c code into the toplevel, like a c
> function, a c++ class method, etc.


Sorry, but I do not understand exactly what you mean by "toplevel". C-LINES
puts whatever code you want into the compiled files: C function definition,
a new C type, etc. This C code can then be used from c-inline.

It would be nice if you could elaborate with examples what you would like to
do.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

On Mon, May 3, 2010 at 5:04 AM, Seth Burleigh <seth@...> wrote:

> (1) A C/C++ program wants to call a lisp function with its own data
> structures:
>

These are called "callbacks". All foreign function interfaces for Lisp
support them, including CFFI and ECL. However ECL currently gives them
rather "obscure" names and does not export them.



> (2) Intertwined development/tight integration: if an algorithm needs to
> be implemented in c/c++ for efficiency, it is much more convenient to
> define it in the lisp code than create another c/c++ file
>

CLINES allows you to embed arbitrary C/C++ code in the file. The only thing
you will not get is that the code is produced in an order which is
consistent with the execution of Common Lisp forms. In other words, if you
insert a CLINES statement in a function it will be moved out of it to the
beginning of the generated C file.

Of course, the code would also have to be able to call lisp functions
> defined in the current package in order for this to be effective (or to
> be able to call lisp functions which have been designated as
> 'callbackable', like in gambit-c scheme) [...]
>

Se above.

(3) Programs built in c++ already have a class hierarchy. If you want to
> create a subclass which the c++ program can use, you need to be able to
> directly input toplevel code (i.e. define a class, methods, etc).
>

CLINES
http://ecls.sourceforge.net/new-manual/re13.html

I insist that this is mostly used for headers, but it need not be.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

Lets say i have a file called test.lisp and in it is this code:

(in-package :utils2)

(FFI:C-INLINE NIL
              NIL
              :VOID
              "void add (a b) {
       return a+b;
       }"
              :ONE-LINER
              NIL
              :SIDE-EFFECTS
              T)

If :side-effects is nil, ecl ignores the above statement. So, now what
does it produce in the c code? It places the above function which we
want in the global scope of the c file into the fasl initialization
function:




/*	Compiler: ECL 10.4.2                                          */
/*	Date: 2010/5/3 06:50 (yyyy/mm/dd)                             */
/*	Machine: Linux 2.6.31-20-generic i686                         */
/*	Source: /home/seth/install/test.lisp                          */
#include <ecl/ecl-cmp.h>
#include "/home/seth/install/test.h"
	
#include "/home/seth/install/test.data"
#ifdef __cplusplus
extern "C"
#endif
ECL_DLLEXPORT void init_fas_CODE(cl_object flag)
{ VT1 VLEX1 CLSR1 STCK1
	const cl_env_ptr cl_env_copy = ecl_process_env();
	cl_object value0;
	cl_object *VVtemp;
	if (!FIXNUMP(flag)){
	Cblock=flag;
	#ifndef ECL_DYNAMIC_VV
	flag->cblock.data = VV;
	#endif
	flag->cblock.data_size = VM;
	flag->cblock.temp_data_size = VMtemp;
	flag->cblock.data_text = compiler_data_text;
	flag->cblock.data_text_size = compiler_data_text_size;
	flag->cblock.cfuns_size = compiler_cfuns_size;
	flag->cblock.cfuns = compiler_cfuns;
	flag->cblock.source =
make_constant_base_string("/home/seth/install/test.lisp");
	return;}
	#ifdef ECL_DYNAMIC_VV
	VV = Cblock->cblock.data;
	#endif
	Cblock->cblock.data_text = "@EcLtAg:init_fas_CODE@";
	VVtemp = Cblock->cblock.temp_data;
	(void)si_select_package(_ecl_static_0)    /*  SELECT-PACKAGE  */;
	void add (a b) {
       return a+b;
       }
}


As you can see, add is in the scope of the initialization function, not
in the 'toplevel' of the c file, i.e. outside of initialization
function.

Code which is required to be outside of initialization function includes
global scope c functions and c++ methods. Most everything else, i
believe, can go in the header file.

Woops, i even typed in incorrect code, but the idea is correct. Also, if
it were possible to do this, it would also have to be possible to call
the lisp functions by creating something which maps the lisp function
calls in the c/c++ code to the obfuscated names.

On Mon, 2010-05-03 at 13:39 +0200, Juan Jose Garcia-Ripoll wrote:
> On Mon, May 3, 2010 at 5:04 AM, Seth Burleigh <seth@...> wrote:
>         (1) A C/C++ program wants to call a lisp function with its own
>         data
>         structures:
> 
> 
> These are called "callbacks". All foreign function interfaces for Lisp
> support them, including CFFI and ECL. However ECL currently gives them
> rather "obscure" names and does not export them.
> 
> 
>  
>         (2) Intertwined development/tight integration: if an algorithm
>         needs to
>         be implemented in c/c++ for efficiency, it is much more
>         convenient to
>         define it in the lisp code than create another c/c++ file
> 
> 
> CLINES allows you to embed arbitrary C/C++ code in the file. The only
> thing you will not get is that the code is produced in an order which
> is consistent with the execution of Common Lisp forms. In other words,
> if you insert a CLINES statement in a function it will be moved out of
> it to the beginning of the generated C file.
> 
> 
>         Of course, the code would also have to be able to call lisp
>         functions
>         defined in the current package in order for this to be
>         effective (or to
>         be able to call lisp functions which have been designated as
>         'callbackable', like in gambit-c scheme) [...]
> 
> 
> Se above.
> 
> 
>         (3) Programs built in c++ already have a class hierarchy. If
>         you want to
>         create a subclass which the c++ program can use, you need to
>         be able to
>         directly input toplevel code (i.e. define a class, methods,
>         etc).
> 
> 
> CLINES
> http://ecls.sourceforge.net/new-manual/re13.html
> 
> 
> I insist that this is mostly used for headers, but it need not be.
> 
> 
> Juanjo
> 
> 
> -- 
> Instituto de Física Fundamental, CSIC
> c/ Serrano, 113b, Madrid 28006 (Spain) 
> http://tream.dreamhosters.com

I wasn't thinking of callbacks per se. Jus being able to define a c
function which takes c types which will then convert the c tyeps to lisp
types and then call the appropriate lisp function.

I imagine some macro like this

(defc (test :int) ((structure1 a) (void* e) (int b) (double c))
  ;;do something)

which would expand to code like this:
(progn
	(cin 
	"
	int test (structure1 a, void* e, int b , double c) {
  	//do type checking, interconvert, etc...
  	//...
  	//call lisp function 
  	cl_object result = gen_Test(altered_a,altere_e,....);
        return TO_FIXNUM(result);
	}")

	(defun gen_Test ...)
        (ffi:clines "//export test function"))

Or i could imagine a macro which allows me to define a unique c++ class,
by converting the class into c++ code, compiling it, and then i can use
the class in any inlined c++ code in the interpreter. If thats possible.

On Mon, May 3, 2010 at 1:54 PM, Seth Burleigh <seth@...> wrote:

> Lets say i have a file called test.lisp and in it is this code:
>
> (in-package :utils2)
>
> (FFI:C-INLINE NIL ..
>

No!!! C-INLINE is for INLINE statements that perform computations in C. NOT
FOR DECLARATIONS.

> (si:system "cat foo.lisp")
(FFI:CLINES "
int foo(int a, int b)
{
return a+b;
}
")

(let ((x (ffi:c-inline (1 2) (:int :int) :int "foo(#0,#1)" :one-liner t)))
  (print x))

> (compile-file "foo" :load t)
;; Loading "/Users/jjgarcia/foo.fas"

3

Is there anything else I am missing here?

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

ffi:clines introduces all code into the header file and does not inline
code into the c file. Can you declare c++ class methods in header files
and in general arbitrary c/c++ code? Can  you call obfuscated lisp
functions in the header file and are functions/classes defined in the
header file automatically exported (which isn't desirable)? (i dont
know, i dont usually do this)

But besides that, i tested your code and was wondering how i access the
c code from the REPL.

> (si:system "cat foo.lisp") (FFI:CLINES " int foo(int a, int b)
{ return a+b; }
")

;;compile file foo and then ....
(load "foo.fas")


(defun test () (let ((x (ffi:c-inline (1 2) (:int :int) :int
"foo(#0,#1)" :one-liner t)))
  (print x)))

TEST
UTILS2> (compile 'test)

;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;;
;;; End of Pass 1.
;;; Note:
;;;   Invoking external command:
;;;   g++ "-I/usr/local/include/"  -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64
-g -O2 -fPIC  -Dlinux -O -w -c "/tmp/ECL0028i12YJ.c" -o
"/tmp/ECL0028i12YJ.o"
/tmp/ECL0028i12YJ.c: In function ‘cl_lispunion* L1test()’:
/tmp/ECL0028i12YJ.c:16: error: ‘foo’ was not declared in this scope
;;; 
;;; Internal error:
;;;   ** (SYSTEM "g++ \"-I/usr/local/include/\"  -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O2 -fPIC  -Dlinux -O -w -c
\"/tmp/ECL0028i12YJ.c\" -o \"/tmp/ECL0028i12YJ.o\"") returned non-zero
value 1
TEST
NIL
NIL

On Mon, May 3, 2010 at 3:03 PM, Seth Burleigh <seth@...> wrote:

> ffi:clines introduces all code into the header file and does not inline
> code into the c file. Can you declare c++ class methods in header files
> and in general arbitrary c/c++ code?


Header files in C/C++ are nothing special at all. The C/C++ preprocessor
will at some point find the #include statement and read in all of the
header. That's all.. The reason why the code is placed there is because it
has to appear before _any_ common lisp code.


> Can  you call obfuscated lisp
> functions in the header file and are functions/classes defined in the
> header file automatically exported (which isn't desirable)? (i dont
> know, i dont usually do this)
>

Forget about "headers" or not "headers". Just think about C/C++ code. Such
code can or can not export things and that does not depend on where you
place it, but rather how you write it. Types are NEVER exported. Functions
may be and in particular the previous example should have read

static int foo(int a, int b) { ....

so that the function does not get exported.


> But besides that, i tested your code and was wondering how i access the
> c code from the REPL.


This is not possible because we are no C/C++ interpreter. You need to
compile and load the compiled files first.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

> 
>         But besides that, i tested your code and was wondering how i
>         access the
>         c code from the REPL.
> 
> 
> This is not possible because we are no C/C++ interpreter. You need to
> compile and load the compiled files first.
> 

I did compile it. I compiled foo.lisp and loaded it. Then i defined the
function test on the repl which used the c function defined in foo.lisp
and then compiled that. It said that the function wasn't declared in the
current scope. I do not know how i would export the c functions which
are defined so i can use them in compiled code that i produce on the
repl, or in other lisp files.

foo.lisp:
(FFI:CLINES "
int foo(int a, int b)
{
return a+b;
}
")

REPL (after loading foo.fas):


UTILS2> (defun test () 
  (let ((x (ffi:c-inline (1 2) (:int :int) :int "foo(#0,#1)" :one-liner
t)))
    (print x)))

TEST
UTILS2> (compile 'test)

;;; OPTIMIZE levels: Safety=2, Space=0, Speed=3, Debug=0
;;;
;;; End of Pass 1.
;;; Note:
;;;   Invoking external command:
;;;   g++ "-I/usr/local/include/"  -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64
-g -O2 -fPIC  -Dlinux -O -w -c "/tmp/ECL0055CnScz.c" -o
"/tmp/ECL0055CnScz.o"
/tmp/ECL0055CnScz.c: In function ‘cl_lispunion* L1test()’:
/tmp/ECL0055CnScz.c:16: error: ‘foo’ was not declared in this scope
;;; 
;;; Internal error:
;;;   ** (SYSTEM "g++ \"-I/usr/local/include/\"  -D_GNU_SOURCE
-D_FILE_OFFSET_BITS=64 -g -O2 -fPIC  -Dlinux -O -w -c
\"/tmp/ECL0055CnScz.c\" -o \"/tmp/ECL0055CnScz.o\"") returned non-zero
value 1
TEST
NIL
NIL

On Mon, May 3, 2010 at 3:44 PM, Seth Burleigh <seth@...> wrote:

> > This is not possible because we are no C/C++ interpreter. You need to
> > compile and load the compiled files first.
> >
>
> I did compile it. I compiled foo.lisp and loaded it.
> UTILS2> (defun test ()
>   (let ((x (ffi:c-inline (1 2) (:int :int) :int "foo(#0,#1)" :one-liner
> t)))
>    (print x)))
>
> TEST
> UTILS2> (compile 'test)
> "/tmp/ECL0055CnScz.o"
> /tmp/ECL0055CnScz.c: In function ‘cl_lispunion* L1test()’:
> /tmp/ECL0055CnScz.c:16: error: ‘foo’ was not declared in this scope


This is a limitation of the compiler and linker. The function "foo" is
defined and loaded in memory, but we have not told the compiler about this
function and second in some platforms it will be impossible to let the
compile code use the "foo" function which is only in memory and not in a
library. So the message is that this is not portable at all, but in some
platforms it is possible. For instance, this is Mac OS X

>  (defun test () (ffi:clines "extern int foo(int a, int b);")
 (let ((x (ffi:c-inline (1 2) (:int :int) :int "foo(#0,#1)" :one-liner
t)))
   (print x)))

TEST
> (compile 'test)
[...]
Undefined symbols:
  "_foo", referenced from:
      _L1test in ECL001YyIGUB.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
> (setf c:*user-ld-flags* '("-Wl,-undefined,dynamic_lookup"))

("-Wl,-undefined,dynamic_lookup")
> (compile 'test)
[...]
> (test)

3
3



-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

>         
> 
> This is a limitation of the compiler and linker. The function "foo" is
> defined and loaded in memory, but we have not told the compiler about
> this function and second in some platforms it will be impossible to
> let the compile code use the "foo" function which is only in memory
> and not in a library. So the message is that this is not portable at
> all, but in some platforms it is possible. For instance, this is Mac
> OS X

How is one supposed to tell the compiler about the function, or in
general any class, class method, structure, etc?


Also, i recall reading somewhere that now fasl files are implemented as
linked libraries? So if you made a fasl file and loaded that?

On Mon, May 3, 2010 at 4:05 PM, Seth Burleigh <seth@...> wrote:

> How is one supposed to tell the compiler about the function, or in
> general any class, class method, structure, etc?
>

Normally one has a C/C++ library with headers that one includes in the
source. The compiler reads the headers (because you have included them) and
then infers the types and functions that are defined in the library. Then
the library is passed to ECL as an argument (see BUILDER description in the
manual) and the linker makes the association such that the compiled code can
call the function

What you intended could not work because your function foo() only existed in
memory. There was no header that could be reused by other files and no file
to be linked against.

What my previous messages implied was the following:

* If you are prototyping classes and functions then define them in the same
file where they are going to be used. Compile, load and that is all.

* If the use your prototypes is going to span multiple lisp files then
create a C/C++ library. Compile and link the FASL files against it and
you're done.

* If you do not feel comfortable with C/C++ please use CFFI and follow the
guidelines there. C-INLINE and CLINES imply some knowledge of how linking
and loading and compiling phase are implemented. CFFI has a different
workflow in which it starts from previously built shared libraries and
everything is simpler that way.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

> 
> * If the use your prototypes is going to span multiple lisp files then
> create a C/C++ library. Compile and link the FASL files against it and
> you're done.

The goal is to incorporate all the c/c++ code  into the lisp files, not
to create a c/c++ library. As i said, tight integration. The c++ library
should be able to call the lisp code, and the lisp code should be able
to call the c++ code. This is certainly possible but involves compiling
the library to a library file each time a change is made, and then you
have to worry about initializing the ecl compiler, initializing the
packages so you can use them in the c++ library, etc. In other words, a
task which should be simple has just been made much more difficult. What
should be an integrated experience has just become a separate one, one
on the c++ side and one on the lisp side.

>Normally one has a C/C++ library with headers ...

I know this, though i dont know much more. I guess i should rephrase my
quesiton. How does ecl do this:

The code below was created from the lisp code:
(defun test2 ()
 (ffi:c-inline nil nil :void "
   CRectangle rect;
   rect.set_values (3,4);
   cout << \"area: \" << rect.area();"
))

How is the below function 'used' by the ecl compiler so that i can load
'foo.fas' from the repl and access the lisp function 'test2'? If the ecl
compiler can create a c function which it tells the compiler about and
which can then be loaded, can i somehow hack the ecl internals so that i
can do the same for my own c function?

static cl_object L1test()
{ VT2 VLEX2 CLSR2 STCK2
	const cl_env_ptr cl_env_copy = ecl_process_env();
	cl_object value0;
	ecl_cs_check(cl_env_copy,value0);
	{
TTL:
	{int V1;                                  /*  X               */
	V1= foo(((int)1),((int)2));
	value0=ecl_print(ecl_make_int(V1),Cnil); cl_env_copy->nvalues=1;
	return value0;
	}
}}

Well, i should probably read something more in depth about linking and
object files. Just looking at foo.fas, i can see the function foo and
test2 in the 'T' section, the code section. I'll read up more and then
see about this.... Thanks for all the help.

On Mon, May 3, 2010 at 4:56 PM, Seth Burleigh <seth@...> wrote:

> The goal is to incorporate all the c/c++ code  into the lisp files, not
> to create a c/c++ library. As i said, tight integration. The c++ library
> should be able to call the lisp code, and the lisp code should be able
> to call the c++ code.
>

Nobody says it can not be done right now. I think right now you have two
conceptual problems.

The first one is that you are assuming that ECL's compiler knows anything
about C. It does not. It simply passes the chunks of code you write to the C
compiler.

The second one is that you want seamless integration with the C world but
the way you conceive it is not the way the C world expects it.

You have to understand the following: a C compiler needs to know what
functions exist, what types there are, etc. This is all fine when one just
writes ONE single lisp file because everything is translated into the SAME
set of C sources.

What happens if you have multiple sources comprising a C application? They
have to *share* the knowledge about the existence of functions and
datatypes. Well, this is done in the header file, which is included in the
sources via an #include and then used.

But what happens when you insist on defining all the C/C++ datatypes and
functions in the lisp files? You *STILL* need a header. You have to have
somewhere a file which declares what you are compiling. Why? Because ECL is
not able to parse that information from your arbitrary chunks of C code.
Otherwise we would be writing a C compiler ourselves.

Now the message is as follows: if you want multiple compiled lisp files to
use the same datatypes and functions, then create a common header, foo.h,
and store there the declarations of the datatypes.

>Normally one has a C/C++ library with headers ...
>
> I know this, though i dont know much more.


As you mentioned in your other email I guess it is important for you to
really learn a bit more about C/C++ if you want to interact with it.


> I guess i should rephrase my
> quesiton. How does ecl do this:
>
> The code below was created from the lisp code:
> (defun test2 ()
>  (ffi:c-inline nil nil :void "
>   CRectangle rect;
>   rect.set_values (3,4);
>   cout << \"area: \" << rect.area();"
> ))
>
> How is the below function 'used' by the ecl compiler


Again, the ECL compiler does not USE the code. It passes it to the C
compiler which in turn produces what is expected. ECL is not going to parse
your structure definitions, it is also not going to find out the C functions
you define in the file and magically export them, it does not attempt to
substitute the C compiler. So when the C compiler gets the code that ECL is
producing it needs some hints.

One way that should just work:

* Create a common header myheader.h
* In that header write all common types you are going to need and declare
the functions that are going to be defined.
* Let all your lisp files include that header using (ffi:clines "#include
\"myheader.h\"") and then, in the lisp files you wish, write the C/C+
_definitions_
* Compile and link everything together using ECL's BUILDER routine. Roughly

(compile-file "file1.lisp" :output-file "file1.o" :system-p t)
...
(compile-file "file13.lisp" :output-file "file1.o" :system-p t)

(c::build-fasl "myfasl" :lisp-files '("file1.o" ... "file13.o"))

[ there may be typos above ]

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

On Mon, 03 May 2010 06:57:29 -0500
Seth Burleigh <seth@...> wrote:

> Woops, i even typed in incorrect code, but the idea is correct. Also, if
> it were possible to do this, it would also have to be possible to call
> the lisp functions by creating something which maps the lisp function
> calls in the c/c++ code to the obfuscated names.

Note that the obfuscated C symbols generated from Lisp code does not
prevent C code from invoking them; i.e. it's possible to use from C
code: cl_object obj = funcall(nargs, @functionsymbol, ...);
-- 
Matt

On Mon, May 3, 2010 at 6:03 PM, Matthew Mondor <mm_lists@...:

> On Mon, 03 May 2010 06:57:29 -0500
> Seth Burleigh <seth@...> wrote:
>
> > Woops, i even typed in incorrect code, but the idea is correct. Also, if
> > it were possible to do this, it would also have to be possible to call
> > the lisp functions by creating something which maps the lisp function
> > calls in the c/c++ code to the obfuscated names.
>
> Note that the obfuscated C symbols generated from Lisp code does not
> prevent C code from invoking them; i.e. it's possible to use from C
> code: cl_object obj = funcall(nargs, @functionsymbol, ...);


Be careful because Seth stated that he does not want to call the C functions
that are used for lisp. He wants to call functions that take as arguments
ordinary C types (structures, pointers, integers, etc) and which
automatically convert the arguments of these functions to lisp, that acting
as proxy for the real lisp functions. That is the definition of a callback.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

> Be careful because Seth stated that he does not want to call the C
> functions that are used for lisp. He wants to call functions that take
> as arguments ordinary C types (structures, pointers, integers, etc)
> and which automatically convert the arguments of these functions to
> lisp, that acting as proxy for the real lisp functions. That is the
> definition of a callback.
> 
> 

So, I guesssing that there isn't a simple parser which replaces the
funcall of the non-obfuscated lisp name with the actual obfuscated
function call? Instead literally the lisp 'funcall' is called? Some
other mechanism i should look up since i need to be able to call the
lisp functions.


--------------------------

Looking at the code generated, I gather/speculate that this is the way
that the functions are 'used' by ecl:


The header file defines an array which contains function pointers
(cl_objectfn cast?)

static const struct ecl_cfun compiler_cfuns[] = {
 /*t,m,narg,padding,name,block,entry*/
{0,0,0,0,MAKE_FIXNUM(2),MAKE_FIXNUM(0),(cl_objectfn)L1test,Cnil,MAKE_FIXNUM(326)},
{0,0,0,0,MAKE_FIXNUM(3),MAKE_FIXNUM(1),(cl_objectfn)L2test2,Cnil,MAKE_FIXNUM(437)},
};

then when we 'load' a fas we're actually loading a shared library, we
call init_FAS_code and pass in a flag object, the flag object will then
contain the compiler_cfuns array which contains the function pointers
which can then be used to access the functions. Is this the way it
happens?

---------------------------------------------------------------------------------------
#ifdef __cplusplus
extern "C"
#endif
ECL_DLLEXPORT void init_fas_CODE(cl_object flag)
{ VT1 VLEX1 CLSR1 STCK1
	const cl_env_ptr cl_env_copy = ecl_process_env();
	cl_object value0;
	cl_object *VVtemp;
	if (!FIXNUMP(flag)){
	Cblock=flag;
	#ifndef ECL_DYNAMIC_VV
	flag->cblock.data = VV;
	#endif
	flag->cblock.data_size = VM;
	flag->cblock.temp_data_size = VMtemp;
	flag->cblock.data_text = compiler_data_text;
	flag->cblock.data_text_size = compiler_data_text_size;
	flag->cblock.cfuns_size = compiler_cfuns_size;
	flag->cblock.cfuns = compiler_cfuns;
	flag->cblock.source =
make_constant_base_string("/home/seth/install/test.lisp");
	return;}
	#ifdef ECL_DYNAMIC_VV
	VV = Cblock->cblock.data;
	#endif
	Cblock->cblock.data_text = "@EcLtAg:init_fas_CODE@";
	VVtemp = Cblock->cblock.temp_data;
	(void)si_select_package(_ecl_static_0)    /*  SELECT-PACKAGE  */;
	ecl_cmp_defun(VV[2]);
	ecl_cmp_defun(VV[3]);
}


---------------------------------------------------------------------------------------

So, now conceptually my ideal .fas would also store an .h file which
contains all the function definitions and structure, class defs, etc per
package scope (lets not worry how we get those, im thinking of clang as
it is a structured compiler with an api).

So, lets now say that when we use a package we can also specify c
functions which have been defined in that package to 'import' into a new
package. When imported, an h file would be imported also consisting of
pieces of the .h file which correspond to what we requested. Than, any
compile-file in that package would piece together all those .h files and
use them, plus also link to the corresponding .fas files.

So, basically if this is conceptually sound what would be needed from
the ecl compiler is a bunch of compiler hooks and loading hooks.

Would this work or am i missing something?


In this way, we could treat c functions, c classes, etc as regular
imports (would be interesting) and everything would be pretty seamlessly
integrated.

On Mon, May 3, 2010 at 7:22 PM, Seth Burleigh <seth@...> wrote:

> So, I guesssing that there isn't a simple parser which replaces the
> funcall of the non-obfuscated lisp name with the actual obfuscated
> function call? Instead literally the lisp 'funcall' is called? Some
> other mechanism i should look up since i need to be able to call the
> lisp functions.
>

I am a bit confused by the jump of requirements. But let me explain the
following: each lisp function gets compiled to a C function with one of
these signatures

cl_object f1(cl_narg narg, cl_object arg1, ...)

cl_object f2(cl_object arg1, cl_object arg2, ..., cl_object argn)

In the first case the "..." are actual optional arguments (this happens for
&key, &optional and &rest functions) and in the second case it is a lisp
function with N arguments.

In principle you could call these functions directly, but we obfuscate the
names on purpose so as to have  the functions not collide with each other
(in CL you may define the same function twice and you are happy, in C it is
not possible).

What Matthew showed was that it is possible to use FUNCALL + a pointer to
the symbol naming the function to call that function. In addition to this
there is a mechanism to actually *name* the function, that is specifying to
ECL what C name the function must have. In this case the function will be
exported and you can call it.


> then when we 'load' a fas we're actually loading a shared library, we
> call init_FAS_code and pass in a flag object, the flag object will then
> contain the compiler_cfuns array which contains the function pointers
> which can then be used to access the functions. Is this the way it
> happens?
>

More or less so. For each pointer the Common Lisp creates a CFUN object
(compiled-function in the Common Lisp type system) which keeps the pointer,
the name and the number of arguments it takes. This object is a Common Lisp
one and can be used by FUNCALL. It is then stored in the appropriate slot of
a symbol so that you can also use the symbol instead in FUNCALL.


> So, now conceptually my ideal .fas would also store an .h file which
> contains all the function definitions and structure, class defs, etc per
> package scope (lets not worry how we get those, im thinking of clang as
> it is a structured compiler with an api).
>

Well, getting CLang to do that is not simpler than embedding CLang in ECL
and using it as a compiler instead. Then you can forget about the rest of
your problems because all definitions will be available to you. Furthermore
by that price you would get ECL with an embedded compiler that can be
distributed anywhere.

So, again, if you turn ECL into a C compiler then you can do what you want,
but this is no little feat. Otherwise there are way simpler ways to live
with C -- and actually I do not see the point in creating an infrastructure
where your C code is so tightly packed with the Common Lisp world: it is not
reusable at all and by that price you might be working entirely in Common
Lisp using FFI.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com

> Well, getting CLang to do that is not simpler than embedding CLang in
> ECL and using it as a compiler instead. Then you can forget about the
> rest of your problems because all definitions will be available to
> you. Furthermore by that price you would get ECL with an embedded
> compiler that can be distributed anywhere.

Thanks for the insights. One last question, ive read somewhere on the
mailing list that you have looked at using llvm as the compiler, or
maybe you were planning to directly use clang?

How did that go and is there anywhere i can get that code or is it still
a work in progress?

On Mon, May 3, 2010 at 8:09 PM, Seth Burleigh <seth@...> wrote:

> Thanks for the insights. One last question, ive read somewhere on the
> mailing list that you have looked at using llvm as the compiler, or
> maybe you were planning to directly use clang?


The code is still work in progress. You find it in ECL's repository under
src/new-cmp. I am right now refactoring it so that both compilers share more
code and I can make further progress, but the new compiler is not yet able
to build itself.

The goal is not to use clang, but rather to first have a different
intermediate representation, based on static-single-assignment in a kind of
lispy language (but imperative, not functional) and then have different
backends: one based on what we do now, another one that uses llvm's SSA
compiler for the llvm virtual machine to produce code, another one that
targets our current bytecodes interpreter (or some improved version of
it...)

The other task, embedding clang, would be orthogonal to that and could be
reused.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28006 (Spain)
http://tream.dreamhosters.com