Embeddable Common-Lisp

> ECL's startup time is unacceptably big for me (on target machine it
> starts for about a minute between shell command and REPL prompt,

I have fixed this to a great extent. In CVS, the code takes between 150
and 350 ms in my slow laptop to start up.

> I couldn't figure out how could I decrease startup time -- I thought
> about using some pre-initialized data hard-coded into fasls instead of
> (read) and re-constructing them each time but at the moment I can't
> figure out how could I do this.

The problem is that what takes the most time is the creation of classes.
Currently we have no way to externalize classes, nor methods (this would
be indeed difficult, given that they should point to the binary code).
However most of the time was spent in the subtypep routine. With the
latest improvements this time is greatly reduced.
>
> Do you have any tips on what should be done to speed the
> initialisation up, how may I start?  Or maybe this kind of
> initialization is unavoidable and I should rather focus on adapting
> some checkpointing code and just restore the running process?

Try building a profiled image using "-p -pg" in CFLAGS and LDFLAGS. This
gives you an idea of where time is spent. Currently I see no major
bottleneck.

> By the way -- can you give me some guide to compiler code (the proper
> compiler, cmp package) -- what are the pieces, how do they fit
> together?

Maybe as time permits. I am currently off-shore and with slow modem
connection. When I come back I should write something in the Wiki. Please
remind me by tuesday if I did not do so.

Juanjo

Hi,

I can confirm a recent mail from Juanjo: startup time has been =
drastically
reduced in CVS code. ECL starts now almost immediately on my machine
(dual-P4 with HT, 1GB RAM). Before that, startup time was between 0.5s =
to
1s. Congrats.

Michael.

On Tue, 2005-07-26 at 20:49 +0200, Maciek Pasternacki wrote:
> By the way -- can you give me some guide to compiler code (the proper
> compiler, cmp package) -- what are the pieces, how do they fit
> together?

I did not have much time, but I managed to write down something about
the ideas behind the lisp-to-C translator:

http://ecls.sourceforge.net/cgi-bin/view/Main/EclCompiler

The main problem on writing something about the compiler is that,
opposite to the rest of ECL, it has not yet been reorganized into a set
of orthogonal and clearly designed functions.

Regards

Juanjo

On Prickle-Prickle, Confusion 68, 3171 YOLD, Juan Jose Garcia Ripoll wrote:

> I did not have much time, but I managed to write down something about
> the ideas behind the lisp-to-C translator:
>
> http://ecls.sourceforge.net/cgi-bin/view/Main/EclCompiler

Thanks.  One more question: why isn't sysfun.lsp compiled into cmp.fas
but distributed as interpreted Lisp?

-- 
__    Maciek Pasternacki <maciekp@...> [ http://japhy.fnord.org/ ]
`| _   |_\  / { ...to co, że błoto i muł, że jeśli coś, to tylko pół;
,|{-}|}| }\/             rzekł Ktoś, gdy mi się śnił -- bym wstał i żył... }
\/   |____/                                       ( Wojciech Waglewski )  -><-

On Wed, 2005-08-03 at 13:14 +0200, Maciek Pasternacki wrote:
> Thanks.  One more question: why isn't sysfun.lsp compiled into cmp.fas
> but distributed as interpreted Lisp?

Because it takes way less memory. Currently, data which is generated in
a compiled module is _never_ garbage collected unless the associated
module is.

Actually, at some point I even considered distributing the whole of the
compiler as source code. I never found a machine where the interpreted
compiler was too slow, this would provide compilation even when ECL is
statically linked, and it would be much easier to debug this component.

This reminds me, that at some point I also found it a neat idea to
distribute source code in ZIP files and make this a new loadable format
for ECL.

Regards,

Juanjo

On Wed, 2005-08-03 at 13:28 +0200, Juan Jose Garcia Ripoll wrote:
> On Wed, 2005-08-03 at 13:14 +0200, Maciek Pasternacki wrote:
> > Thanks.  One more question: why isn't sysfun.lsp compiled into cmp.fas
> > but distributed as interpreted Lisp?
> 
> Because it takes way less memory. Currently, data which is generated in
> a compiled module is _never_ garbage collected unless the associated
> module is.

Of course, I am referring to constants found in compiled code, not to
arbitrary data that might by produced at run time by the code that was
compiled.

Regards

Juanjo

On Wed, 03 Aug 2005 13:28:42 +0200
Juan Jose Garcia Ripoll <lisp@...> wrote:

> This reminds me, that at some point I also found it a neat idea to
> distribute source code in ZIP files and make this a new loadable
> format for ECL.

Having some kind of quasi-standardized binary format would allow many
cool things. I.e. a Smalltalk-like development environment. But it
would not be very useful if ECL were the only CL implementation to
support this. 

Regards,
-- 
Julian Stecklina

LISP has survived for 21 years because it is an approximate local
optimum in the space of programming languages. - John McCarthy (1980)

Currently, large program compiled by ECL start up visibly slower
than using other Lisps.  My estimate is that during FriCAS bootstrap
main FriCAS binary is started up about 1000 times, each startup taking
about 1s, which adds up to rather substantial 1000s.  Also during
testsuite run FriCAS binary is invoked about 150 times, this time
each invocation costs about 2.8s, for total of order 420s.  Below
some measurement trying to find out reasons.


One case is ECL compiler:

$ (export LD_LIBRARY_PATH=/var/tmp/hebisch/usr/lib; AXIOM=/var/tmp/hebisch/axp5/ax-build1/target/x86_64-unknown-linux; echo "(time (require 'cmp)) (quit)" | time /var/tmp/hebisch/usr/bin/ecl)
ECL (Embeddable Common-Lisp) 0.9j (CVS 2008-05-14 13:43)
Copyright (C) 1984 Taiichi Yuasa and Masami Hagiya
Copyright (C) 1993 Giuseppe Attardi
Copyright (C) 2000 Juan J. Garcia-Ripoll
ECL is free software, and you are welcome to redistribute it
under certain conditions; see file 'Copyright' for details.
Type :h for Help.  Top level.
> ;;; Loading #P"/var/tmp/hebisch/usr/lib/ecl/cmp.fas"
;;; Loading #P"/var/tmp/hebisch/usr/lib/ecl/sysfun.lsp"
real time : 0.257 secs
run time  : 0.220 secs
gc count  : 7 times
consed    : 9652984 bytes
("CMP")
> 0.43user 0.00system 0:00.48elapsed 88%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+1560minor)pagefaults 0swaps

Another reason is that loading large number of fasls takes a lot of time.
>From FriCAS prompt we load 1090 fasls:

(1) -> )lisp (time (dolist (el (directory "ax-build1/target/x86_64-unknown-linux/algebra/*.fas")) (load el)))
real time : 42.731 secs
run time  : 38.810 secs
gc count  : 1091 times
consed    : 56051704 bytes
Value = NIL
(1) -> )lisp (time (si::gc t))
real time : 0.047 secs
run time  : 0.050 secs
gc count  : 2 times
consed    : 0 bytes
Value = NIL
(1) -> )lisp (time (si::gc nil))
real time : 0.047 secs
run time  : 0.050 secs
gc count  : 2 times
consed    : 0 bytes
Value = NIL

So, it seems that loading each fasl triggers garbage collection,
and those collections are responsible for substantial part
of the load time.  Is there a way to make garbage collection
more lazy?

-- 
                              Waldek Hebisch
hebisch@...

On Wed, May 28, 2008 at 2:42 PM, Waldek Hebisch
<hebisch@...> wrote:
> Currently, large program compiled by ECL start up visibly slower
> than using other Lisps.

There is a reason for that. When ECL boots, it has to run through all
the code that loads the lisp library. In other words, it has to repeat
all the DEFUN DEFVAR, statements, etc. This is because there is not a
portable way to dump a memory image of all the required data and
installed C functions. Same thing applies to loading a FASL.

>  My estimate is that during FriCAS bootstrap
> main FriCAS binary is started up about 1000 times, each startup taking
> about 1s, which adds up to rather substantial 1000s.  Also during
> testsuite run FriCAS binary is invoked about 150 times, this time
> each invocation costs about 2.8s, for total of order 420s.  Below
> some measurement trying to find out reasons.

I see. I would like to understand why you need to load FriCAS 1000
times instead of reusing the same binary for building all compiled
files. Either directly, or using Unix fork().

> One case is ECL compiler:

Ok, here the case is not the compiler itself but the database of
profiled functions in sysfun.lsp. Perhaps we can dump it into a
quicker to load form.

> Another reason is that loading large number of fasls takes a lot of time.
> >From FriCAS prompt we load 1090 fasls:

Why don't you join all those fasl files into a single one using c:build?

> So, it seems that loading each fasl triggers garbage collection,
> and those collections are responsible for substantial part
> of the load time.  Is there a way to make garbage collection
> more lazy?

Sure! Remove the si_gc(Ct) statement in src/c/load.d and please tell
me whether this helps. I can remove it from the default configuration.

Juanjo

--
Facultad de Fisicas, Universidad Complutense,
Ciudad Universitaria s/n Madrid 28040 (Spain)
http://juanjose.garciaripoll.googlepages.com

Juan Jose Garcia-Ripoll wrote:
> On Wed, May 28, 2008 at 2:42 PM, Waldek Hebisch
> <hebisch@...> wrote:
> > Currently, large program compiled by ECL start up visibly slower
> > than using other Lisps.
> 
> > Another reason is that loading large number of fasls takes a lot of time.
> > >From FriCAS prompt we load 1090 fasls:
> 
> Why don't you join all those fasl files into a single one using c:build?
>

I need to load some fasls before generating other ones, so I would
have to keep both .o files and fasls.  That would considerably
complicate build process and increase disc usage.  And during build
I would still have to load fasls.  Also, 1090 (now actually 1107) fasls
is a worst case, in normal runs I need between 100-200 fasls.

> > So, it seems that loading each fasl triggers garbage collection,
> > and those collections are responsible for substantial part
> > of the load time.  Is there a way to make garbage collection
> > more lazy?
> 
> Sure! Remove the si_gc(Ct) statement in src/c/load.d and please tell
> me whether this helps. I can remove it from the default configuration.
> 

I have now tried this modification on ECL 0.9l (CVS 2008-06-19 17:09)
Stock version gives me:

1) -> )lisp (time (dolist (el (directory "target/x86_64-unknown-linux/algebra/*.fas") ) (load el)))
real time : 27.550 secs
run time  : 24.789 secs
gc count  : 1108 times
consed    : 46275376 bytes

Modified version gives:

(1) -> )lisp (time (dolist (el (directory "target/x86_64-unknown-linux/algebra/*.fas") ) (load el)))
real time : 1.323 secs
run time  : 1.192 secs
gc count  : 7 times
consed    : 46293536 bytes

So the speed difference is substantial.

-- 
                              Waldek Hebisch
hebisch@...

On Tue, Oct 14, 2008 at 5:01 AM, Waldek Hebisch
<hebisch@...> wrote:
>> Sure! Remove the si_gc(Ct) statement in src/c/load.d and please tell
>> me whether this helps. I can remove it from the default configuration.
>
> I have now tried this modification on ECL 0.9l (CVS 2008-06-19 17:09)
> Stock version gives me: [...]
> real time : 27.550 secs
> run time  : 24.789 secs
> gc count  : 1108 times
> consed    : 46275376 bytes
>
> Modified version gives:[...]
> real time : 1.323 secs
> run time  : 1.192 secs
> gc count  : 7 times
> consed    : 46293536 bytes
>
> So the speed difference is substantial.

I am glad to read this. This change has been committed to CVS several
months ago, so you should be getting similar speedups with the next
release. If this is not the case, please report it immediately

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28009 (Spain)
http://juanjose.garciaripoll.googlepages.com

Juan Jose Garcia-Ripoll wrote:
> On Tue, Oct 14, 2008 at 5:01 AM, Waldek Hebisch
> <hebisch@...> wrote:
> >> Sure! Remove the si_gc(Ct) statement in src/c/load.d and please tell
> >> me whether this helps. I can remove it from the default configuration.
> >
> > I have now tried this modification on ECL 0.9l (CVS 2008-06-19 17:09)
> > Stock version gives me: [...]
> > real time : 27.550 secs
> > run time  : 24.789 secs
> > gc count  : 1108 times
> > consed    : 46275376 bytes
> >
> > Modified version gives:[...]
> > real time : 1.323 secs
> > run time  : 1.192 secs
> > gc count  : 7 times
> > consed    : 46293536 bytes
> >
> > So the speed difference is substantial.
> 
> I am glad to read this. This change has been committed to CVS several
> months ago, so you should be getting similar speedups with the next
> release. If this is not the case, please report it immediately
> 

With current CVS I still get 25 seconds.

AFAICS the change is:

        /* A full garbage collection enables us to detect unused code
           and leave space for the library to be loaded. This is only
           required when we use the dlopen wrappers. */
-#ifndef GBC_BOEHM_GENGC
-       si_gc(Ct);
-#endif
+       if (!ecl_get_option(ECL_OPT_INCREMENTAL_GC)) {
+               si_gc(Ct);
+       }

so, it seems that the change is effective only I use incremental GC.
However, my configure line is:

./configure --prefix=/var/tmp/hebisch/usr2



-- 
                              Waldek Hebisch
hebisch@...

On Tue, Oct 14, 2008 at 5:09 PM, Waldek Hebisch
<hebisch@...> wrote:
> With current CVS I still get 25 seconds.
>
> AFAICS the change is:
>
>        /* A full garbage collection enables us to detect unused code
>           and leave space for the library to be loaded. This is only
>           required when we use the dlopen wrappers. */
> -#ifndef GBC_BOEHM_GENGC
> -       si_gc(Ct);
> -#endif
> +       if (!ecl_get_option(ECL_OPT_INCREMENTAL_GC)) {
> +               si_gc(Ct);
> +       }
>

Thanks for reporting, I will remove these lines.

Juanjo

-- 
Instituto de Física Fundamental, CSIC
c/ Serrano, 113b, Madrid 28009 (Spain)
http://juanjose.garciaripoll.googlepages.com