From: SourceForge.net <no...@so...> - 2008-03-17 17:46:50
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-17 22:03:36
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-18 17:29:37
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by y-d You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-18 18:31:54
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-18 22:47:26
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by y-d You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 13:56:07
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 15:55:53
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 16:12:14
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 16:19:59
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 17:09:43
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by y-d You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 17:40:57
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by y-d You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:40 Message: Logged In: YES user_id=1993164 Originator: YES > TheSbvector(as_object(0)) This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-19 18:16:46
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-19 14:15 Message: Logged In: YES user_id=5735 Originator: NO >> TheSbvector(as_object(0)) >This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec where? I don't understand, sorry. ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:40 Message: Logged In: YES user_id=1993164 Originator: YES > TheSbvector(as_object(0)) This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-20 15:42:26
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by y-d You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-20 11:42 Message: Logged In: YES user_id=1993164 Originator: YES Well, one example is in '#define jitc_funcallc' on line 711. I need to call interpret_bytecode_, but it requires TheSbvector(vec) as second argument, not just vec. And I can't just use the predefined macro because vec is not available at compile time. As said in lispbibl.c:3385, all 'The##Type' does in HEAPCODES is substract a 'bias' to the argument. So I do that at runtime. Example(lightning.c:736): // JIT_R0 contains address of a closure // Add (the offset of field 'clos_codevec' in Cclosure minus the bias for TheCclosure) to JIT_R0, load the content at this address in JIT_R1 jit_ldxi_p(JIT_R1, JIT_R0, TheCclosure(as_object(jit_ptr_field(Cclosure, clos_codevec)))); // JIT_R1 contains the address of the closure's codevec // Add it to TheSbvector(as_object(0)), which is equal to -varobject_bias jit_addi_p(JIT_R1,JIT_R1,TheSbvector(as_object(0))); Why not just use '-varobject_bias'? I thought using the macro was better because '-varobject_bias' is obscure(it's never used, and people might not know that it's just the def of TheSbvector). ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 14:15 Message: Logged In: YES user_id=5735 Originator: NO >> TheSbvector(as_object(0)) >This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec where? I don't understand, sorry. ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:40 Message: Logged In: YES user_id=1993164 Originator: YES > TheSbvector(as_object(0)) This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-03-20 15:59:13
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Comment added) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) Assigned to: Bruno Haible (haible) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-03-20 11:59 Message: Logged In: YES user_id=5735 Originator: NO why are you calling interpret_bytecode_? (that was a question I wanted to ask for a long time!) I think you want to call its jitc counterpart interpret_bytecode defined around eval.d:318 (should be turned into a function for your purposes, I can do that for you) so you are saying that TheSbvector(as_object(0)) is just a trick to extract byteptr_in from codeptr at run time? that should be fixed by the aforementioned function, right? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-20 11:42 Message: Logged In: YES user_id=1993164 Originator: YES Well, one example is in '#define jitc_funcallc' on line 711. I need to call interpret_bytecode_, but it requires TheSbvector(vec) as second argument, not just vec. And I can't just use the predefined macro because vec is not available at compile time. As said in lispbibl.c:3385, all 'The##Type' does in HEAPCODES is substract a 'bias' to the argument. So I do that at runtime. Example(lightning.c:736): // JIT_R0 contains address of a closure // Add (the offset of field 'clos_codevec' in Cclosure minus the bias for TheCclosure) to JIT_R0, load the content at this address in JIT_R1 jit_ldxi_p(JIT_R1, JIT_R0, TheCclosure(as_object(jit_ptr_field(Cclosure, clos_codevec)))); // JIT_R1 contains the address of the closure's codevec // Add it to TheSbvector(as_object(0)), which is equal to -varobject_bias jit_addi_p(JIT_R1,JIT_R1,TheSbvector(as_object(0))); Why not just use '-varobject_bias'? I thought using the macro was better because '-varobject_bias' is obscure(it's never used, and people might not know that it's just the def of TheSbvector). ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 14:15 Message: Logged In: YES user_id=5735 Originator: NO >> TheSbvector(as_object(0)) >This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec where? I don't understand, sorry. ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:40 Message: Logged In: YES user_id=1993164 Originator: YES > TheSbvector(as_object(0)) This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |
From: SourceForge.net <no...@so...> - 2008-05-25 20:21:54
|
Patches item #1880809, was opened at 2008-01-27 13:25 Message generated for change (Settings changed) made by sds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Yann Nicolas Dauphin (y-d) >Assigned to: Sam Steingold (sds) Summary: JIT via lightning Initial Comment: Still incomplete JIT compiler for CLISP. The jit compilation occurs at every function call(no reuse), so it runs slower for now. Lightning is used because: - It's portable. Can reportedly be ported under a day. - It's simple. If the GNU lightning project was to stop, it wouldn't be a problem. Maintaining it is easy. Not so for LLVM. - It's lean and mean. Code generation is straightforward, so very fast. Tested on x86 with: - Linux - Mac OS X: Must 'export DYLD_BIND_AT_LAUNCH=' for now because the dynamic linker messes with jit function calls The patch is too big for SF so: http://www.step.polymtl.ca/~polyrad/polyrad/jit-clisp-2.43.patch ---------------------------------------------------------------------- >Comment By: Sam Steingold (sds) Date: 2008-05-25 16:21 Message: Logged In: YES user_id=5735 Originator: NO please open a new patch item for further patches. this is already long enough... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-20 11:59 Message: Logged In: YES user_id=5735 Originator: NO why are you calling interpret_bytecode_? (that was a question I wanted to ask for a long time!) I think you want to call its jitc counterpart interpret_bytecode defined around eval.d:318 (should be turned into a function for your purposes, I can do that for you) so you are saying that TheSbvector(as_object(0)) is just a trick to extract byteptr_in from codeptr at run time? that should be fixed by the aforementioned function, right? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-20 11:42 Message: Logged In: YES user_id=1993164 Originator: YES Well, one example is in '#define jitc_funcallc' on line 711. I need to call interpret_bytecode_, but it requires TheSbvector(vec) as second argument, not just vec. And I can't just use the predefined macro because vec is not available at compile time. As said in lispbibl.c:3385, all 'The##Type' does in HEAPCODES is substract a 'bias' to the argument. So I do that at runtime. Example(lightning.c:736): // JIT_R0 contains address of a closure // Add (the offset of field 'clos_codevec' in Cclosure minus the bias for TheCclosure) to JIT_R0, load the content at this address in JIT_R1 jit_ldxi_p(JIT_R1, JIT_R0, TheCclosure(as_object(jit_ptr_field(Cclosure, clos_codevec)))); // JIT_R1 contains the address of the closure's codevec // Add it to TheSbvector(as_object(0)), which is equal to -varobject_bias jit_addi_p(JIT_R1,JIT_R1,TheSbvector(as_object(0))); Why not just use '-varobject_bias'? I thought using the macro was better because '-varobject_bias' is obscure(it's never used, and people might not know that it's just the def of TheSbvector). ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 14:15 Message: Logged In: YES user_id=5735 Originator: NO >> TheSbvector(as_object(0)) >This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec where? I don't understand, sorry. ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:40 Message: Logged In: YES user_id=1993164 Originator: YES > TheSbvector(as_object(0)) This is used when calling interpret_bytecode_ to add the 'bias' to the address of codevec ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-19 13:09 Message: Logged In: YES user_id=1993164 Originator: YES on it ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:19 Message: Logged In: YES user_id=5735 Originator: NO what is TheSbvector(as_object(0))? as_object(0)==nullobj why are you casting it to TheSbvector? this is confusing... ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 12:08 Message: Logged In: YES user_id=5735 Originator: NO the aforementioned crash is in the first call to bc_func in jitc_run ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 11:45 Message: Logged In: YES user_id=5735 Originator: NO jitc_get_fieldr causes "warning: cast from pointer to integer of different size" on amd64 with SAFETY=3 also, how about re-using S_operand_ignore et al from eval.d? what I meant by the previous comment is that you do not need aclocal et al - just pass --disable-maintainer-mode to the top-level configure script. I now get to ... ;; Loading file /homedata/ssteingold/src/clisp/current/src/defs2.lisp ... Program received signal SIGSEGV, Segmentation fault. 0x00000000008f0db3 in ?? () (gdb) where #0 0x00000000008f0db3 in ?? () #1 0x00000000008f10c5 in ?? () #2 0x000000099a35e6d9 in ?? () #3 0x00000000008ec7b0 in ?? () #4 0x000000099a35e6d9 in ?? () #5 0x00000000008f2f49 in ?? () #6 0x00000002008c8421 in ?? () #7 0x000000099a2f19b3 in ?? () #8 0xffffff6f008c84a1 in ?? () #9 0x00000000008f2f4e in ?? () #10 0x00000000008f10c5 in ?? () #11 0x000000099a35e6f0 in ?? () #12 0x00007fff18a96a40 in ?? () #13 0x0000000000411390 in allocate_fpointer (foreign=0x7fff18a96a40) at ../src/spvw_typealloc.d:456 #14 0x00007fff18ae1880 in ?? () #15 0x0000000000000000 in ?? () (gdb) ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-19 09:56 Message: Logged In: YES user_id=5735 Originator: NO http://clisp.cons.org/impnotes/faq.html#faq-autoconf ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 18:47 Message: Logged In: YES user_id=1993164 Originator: YES Thanks a lot. I subscribed right away. Unfortunately, I cannot compile clisp on those servers. A lot of important programs are missing(aclocal for one). Do you know any other service like this one? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-18 14:31 Message: Logged In: YES user_id=5735 Originator: NO USE_JITC requires HEAPCODES -- OK, I am now on HEAPCODES. still on 64-bits I get plenty of "cast from pointer to integer of different size". if you need access to a 64-bit system: http://www.testdrive.hp.com/current.shtml this looks bad: lightning.c:1714:24: error: macro "jit_qop_" requires 5 arguments, but only 4 given lightning.c:1714: error: 'jit_qop_' undeclared (first use in this function) lightning.c:1714: error: (Each undeclared identifier is reported only once lightning.c:1714: error: for each function it appears in.) also, how about re-using S_operand_ignore et al from eval.d? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-18 13:29 Message: Logged In: YES user_id=1993164 Originator: YES Typecodes and Heapcodes: I failed to mention it, but right now only Heapcodes is supported. I chose not to support both because that would require a lot of code duplication(it's assembly). That wouldn't be a good idea for an initial release. It would make everything harder. I chose Heapcodes because it's supported on every platform, it's as fast(said in imp notes), it keeps the code simple, and because I cannot use Typecodes(I have a 32 bit system). Of course, when jitc has been tested and is functionnal with Heapcodes, we can support typecodes. Ciao ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 18:03 Message: Logged In: YES user_id=5735 Originator: NO jitc_finish_framer is defined incorrectly: its definition is HEAPCODES-specific and does not work for TYPECODES. it has to be redefined to use framebottomword instead of makebottomword. same for cons_bias and tfl - they have no place in lightning.c ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-17 13:46 Message: Logged In: YES user_id=5735 Originator: NO the cast does not help (as expected - the same expression appears in eval.d without a cast). on amd64 we use a different memory model (TYPECODES vs HEAPCODES on i386) which causes a different set of issues. I find quite confusing: lightning.c: In function 'jit_compile_': lightning.c:1300: warning: implicit declaration of function 'LEAQmr' lightning.c:1312: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1346: warning: implicit declaration of function '_s32P' lightning.c:1760: warning: implicit declaration of function 'jitc_getsize_framer' lightning.c:2010: warning: implicit declaration of function 'jit_nei_l' ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-16 23:29 Message: Logged In: YES user_id=1993164 Originator: YES I don't get an error, but I guess the following should work: Change line 5448 of lightning.c to: pushSTACK(fixnum(byteptr - (const uintB*)codeptr->data - byteptr_min)); On a separate matter, change line 170 of spvw_gcmark. to: if (cclosurep(curr) && fpointerp(cclosure_jitc(curr))) gc_mark_jitc_object(TheFpointer(cclosure_jitc(curr))->fp_pointer) Tell me how it works out ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 17:03 Message: Logged In: YES user_id=5735 Originator: NO I get this: In file included from ../src/eval.d:8265: lightning.c: In function 'jit_compile_': lightning.c:5448: error: invalid operands to binary - ideas? ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-16 12:01 Message: Logged In: YES user_id=5735 Originator: NO thanks for the patch. now, there appears to be a lot of assembly copied verbatim from eval.d to lightning.c - is there a way to avoid it? since lightning.c is included by eval.d, maybe all those S_operand_ignore &friends could be re-used? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-15 13:48 Message: Logged In: YES user_id=1993164 Originator: YES This problem has now been fixed. I can successfully make interpreted.mem The problem is now loading it. #> gdb lisp.run (gdb) run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Starting program: /Users/lokee/devel/clisp/clisp/src/lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -M interpreted.mem -q -c compiler.lisp -o ./ Reading symbols for shared libraries .. done Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_PROTECTION_FAILURE at address: 0x00000000 0x00000000 in ?? () (gdb) backtrace #0 0x00000000 in ?? () #1 0x00017275 in jitc_run (closure=0x1a8dbc05, codeptr=0x1a8dbbe8, byteptr_in=0xa <Address 0xa out of bounds>) at lightning.c:5493 ... ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-03-14 23:25 Message: Logged In: YES user_id=1993164 Originator: YES A little update, All clean-ups are done. A few changes to 'jit_compile' allows functions to be jit compiled and then executed directly. Previous errors are now fixed! The problem now is garbage collection: #> gdb lisp.run (gdb) run -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit))" ... ;; Loading file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/compiler.lisp ;; Loading file /Users/lokee/devel/clisp/clisp/src/defs2.lisp ... ;; Loaded file /Users/lokee/devel/clisp/clisp/src/defs2.lisp Program received signal EXC_BAD_ACCESS, Could not access memory. Reason: KERN_INVALID_ADDRESS at address: 0x8019e8e1 0x0001669a in gc_mark_jitc_object (ptr=0x8019e8e1) at lightning.c:73 73 jo->jo_gc_mark = 1; We're close. I'll be investigating on this tomorrow. (See jit-clean-up.patch.gz for reference) Ciao File Added: jit-clean-up.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-03-11 16:02 Message: Logged In: YES user_id=5735 Originator: NO any progress? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-26 16:46 Message: Logged In: YES user_id=1993164 Originator: YES Thanks, I'll make the necessary changes. Some other things are keeping me tied up right now, but I expect to send in a patch somewhere in the next week. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 12:04 Message: Logged In: YES user_id=5735 Originator: NO Paolo: on jit_movi_p returning a value: "yes [it is intentional], you can use the return value to patch a forward reference. if you don't need the value, you can use jit_movi_l, but you will still get warnings from e.g. the branch statements." I guess you have to cast it to (void) in your #defines. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-26 11:41 Message: Logged In: YES user_id=5735 Originator: NO also, it would be nice if your macros were named in a distinct way from the built-in lightning macros. both are named using "jit_" prefix now, and it is confusing. maybe clisp macros should be named "jitc_"? thanks. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 14:42 Message: Logged In: YES user_id=5735 Originator: NO OK, your jit-objstruct.patch.gz patch finally got my attention. OUCH! the cavalier way you handle lisp objects is scary. this is actually my fault, I should have noticed this earlier. sorry... === memory management. you allocate Cons with malloc instead of the CLISP allocate_cons() function. 1. where do you free it? 2. why malloc? if you manage this data yourself and it lives a separate life from the Lisp heap, then there is no reason for you to use CLISP data structure - just define your own pair type. if this is supposed to link into (or be linked to from) the Lisp heap, then, given CLISP copying GC, you are bound to get segfaults. === lisp data field access. you write new_cell->cdr - use TheCdr() instead. you write (list != (Cons)as_oint(NIL)) - use !eq(list,NIL) or !nullp(list) instead thanks! ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-22 13:40 Message: Logged In: YES user_id=5735 Originator: NO >> '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' in that case you should invoke jit_movi_p with a cast: (void)jit_movi_p(x,y) (unless, of course, you patch will be accepted by Paolo Bonzini) ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-17 19:57 Message: Logged In: YES user_id=1993164 Originator: YES I attached a patch for OBJECT_STRUCT support. The 'value computed is not used' happen because some lightning macros return values when they shouldn't. For example jit_movi_p is defined as: '#define jit_movi_p(d, is) (jit_movi_ul((d), (long) (is)), _jit.x.pc)' I don't know why the expression returns '_jit.x.pc' yet. If there's no reason, I'll send a patch to the lightning ML. As for the other errors, what version of lighting do you use? What architecture are you on? What flags are on? When I redefine 'interpret_bytecode' to force jit execution, I get: ./lisp.run -B . -E 1:1 -Efile UTF-8 -Eterminal UTF-8 -norc -m 2MW -lp -x "(and (load \"init.lisp\") (sys::%saveinitmem) (ext::exit)) (ext::exit t)" [...] ;; Loading file /Users/lokee/clisp/src/compiler.lisp ... ;; Loaded file /Users/lokee/clisp/src/compiler.lisp ;; Loading file /Users/lokee/clisp/src/defs2.lisp ...make: *** [interpreted.mem] Segmentation fault Thank you, File Added: jit-objstruct.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-13 18:48 Message: Logged In: YES user_id=5735 Originator: NO OK, so the first stab at keeping JITC code with the closure is in the CVS head. Sorry about taking so long... Alas, I cannot really test it because lightning does not compile for me. what I observe now is when I compile lightning.c I see lots of errors and warnings, e.g., lightning.c: In function 'jit_compile_': lightning.c:1304: warning: implicit declaration of function 'LEAQmr' lightning.c:1304: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1310: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1311: warning: value computed is not used lightning.c:1316: warning: implicit declaration of function 'jit_muli_ul' lightning.c:1316: warning: value computed is not used lightning.c:1324: error: 'cod_labels' undeclared (first use in this function) lightning.c:1324: error: (Each undeclared identifier is reported only once lightning.c:1324: error: for each function it appears in.) lightning.c:1324: error: expected ';' before ':' token lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1338: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1342: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: implicit declaration of function '_s32P' lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: value computed is not used lightning.c:1350: warning: cast from pointer to integer of different size lightning.c:1350: warning: value computed is not used lightning.c:1357: warning: value computed is not used also, you assume that object is a pointer type which it is not when SAFETY is 3 and OBJECT_STRUCT is defined. so, the ball is in your court is now: removing warnings supporting OBJECT_STRUCT testing the GC hooks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 19:03 Message: Logged In: YES user_id=5735 Originator: NO oops - forgot that fpointers are usually immediate. we can have a chain of jit functions and mark them at gc_mark and then sweep them. this is a variation on the finaliser idea. ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-04 00:23 Message: Logged In: YES user_id=5735 Originator: NO then we are in trouble wrt re-using the compiled form. we need to store the compiled closure (codeBuffer + bcIndex &c) in the Cclosure as a Fpointer to a C structure. how do we release it when the Cclosure object is garbage collected? (a) the most direct way is to use finalize, but that would make GC very expensive. (b) we can add a 3rd phase between mark and copy - scan memory for dead cclosures and free the jit code (also expensive) (c) we can allocate Fpointers in a separate memory area and mark those who should be free()d with a special bit, i.e., special case finalizing Fpointers with free(), then the additional phase as in (b) has to be done only on that separate fpointer memory area. note that this feature may also be useful in other areas, e.g., FFI. Bruno, what do you think? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 22:44 Message: Logged In: YES user_id=1993164 Originator: YES codeBuffer and bcIndex have to be kept in constant memory because the operands of jump instructions in the JITed code are absolute positions. As a side-note, I do plan to remove the need to permanently store bcIndex. But that's not high priority. Thanks ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-02-03 21:00 Message: Logged In: YES user_id=5735 Originator: NO thanks - I committed your patch. do codeBuffer & bcIndex have to be in constant memory (as in malloc) as opposed to lisp heap (movable by GC)? ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-02-03 11:22 Message: Logged In: YES user_id=1993164 Originator: YES Lightning vs Libjit Technical: Libjit is well written and looks lovely, but Lightning too. The main differences are that Libjit is more high-level (Auto register allocation, memory allocation, ...) and that it optimizes the code a bit. This is why I first considered using libjit. But [1] shows that an optimizing(as in AOT compilers) compiler working at a low-level is not dramatically faster that a template based JIT compiler(like lightning). For significant performance improvements, using trace based JIT compilation with Trace SSA code analysis as described in [2] is the best alternative. This approach requires control over register allocation and optimizes the code. This makes much of libjit features redundant. Social: Shortly: Lightning is more active. GNU Smalltalk, MzScheme, OcamlJIT, Rubinius(in FFI module) use it. MzScheme achieves good performance with it[3]. No active project uses libjit. Which is a shame, since it's a good lib. The current version of libjit only supports 1 architecture while Lightning supports 4. And it seems that number will keep on growing. It's risky to use Libjit. [1] http://research.sun.com/techrep/2000/abstract-87.html Also note the speed up over a simple switch based interpreter for number crunshing. [2] http://www.usenix.org/events/vee06/full_papers/p144-gal.pdf This has the merit of optimizing 0 (CONST&PUSH 0) ; 2 1 (CONST 1) ; (4) 2 (CONS) So that 0 and 1 don't push anything on the stack and get directly passed in registers (if possible). [3] http://shootout.alioth.debian.org/gp4/benchmark.php?test=all&lang=all ---------------------------------------------------------------------- Comment By: Yann Nicolas Dauphin (y-d) Date: 2008-01-30 01:53 Message: Logged In: YES user_id=1993164 Originator: YES jit-clisp-cvs-20080131.patch.gz: This new patch includes the clean ups from Reini and will work against a clean checkout from CVS. Please install the latest version of GNU lightning(at least 1.2b): http://git.savannah.gnu.org/gitweb/?p=lightning.git;a=snapshot;h=master It includes: - Modifying make file to add jit.d with jit.h as one of it's depedencies - Assuming GNU Lightning is installed - lispbibl sets USE_JIT if I80386 is set and registers are saved if !USE_JIT - Replacing all // comments with /**/ and tabs with spaces - Removed vestigial c source and jit_begin/end - Random cleanups News tests have been added to 'make check'. CLISP runs out of memory on one of the last ones, 'ctak'. This is because using longjmp can cause the memory of codeBuffer to leak. This is of course resolved by reusing the jit function. libjit vs lightning coming up. File Added: jit-clisp-cvs-20080131.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:25 Message: Logged In: YES user_id=5735 Originator: NO clean-up? http://permalink.gmane.org/gmane.lisp.clisp.devel/17523 http://rurban.xarch.at/cygr/clisp/jit-extra.patch.gz ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 18:22 Message: Logged In: YES user_id=5735 Originator: NO lightning vs libjit http://permalink.gmane.org/gmane.lisp.clisp.general/12114 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-28 17:00 Message: Logged In: YES user_id=5735 Originator: NO it would be nice if you could elaborate on advantages of lightning over libjit http://freshmeat.net/projects/libjit/ ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 21:14 Message: Logged In: YES user_id=5735 Originator: NO http://article.gmane.org/gmane.lisp.clisp.devel:17497 http://article.gmane.org/gmane.lisp.clisp.devel:17509 http://article.gmane.org/gmane.lisp.clisp.devel:17515 ---------------------------------------------------------------------- Comment By: Sam Steingold (sds) Date: 2008-01-27 19:52 Message: Logged In: YES user_id=5735 Originator: NO the patch is too big because it includes some binary files (.DS_Store) and the full sources for lightning. please assume that lightning is installed on the developer's machine. please include only the eval.d and jit.h, NOT lightning sources please do NOT use "//" comments ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=301355&aid=1880809&group_id=1355 |