Embeddable Common-Lisp

Nicolas Neuss <Nicolas.Neuss@...> writes:

> 3. Loading my floating point performance test
>
>   <http://sit.iwr.uni-heidelberg.de/~neuss/WWWPages/misc/mflop.lisp>

Should read: <http://sit.iwr.uni-heidelberg.de/~neuss/misc/mflop.lisp>

Nicolas.

On Thu, 2005-08-04 at 18:06 +0200, Nicolas Neuss wrote: 
> I am just evaluating if it is possible to use ECL reasonably for my PDE
> toolbox Femlisp.  My requirements are fast floating point operations and
> quite some ANSI compatibility.

As far as ECL is concerned, floating point numbers are unboxed inside a
function, but not across functions (not yet, at least). Most operations
with these unboxed numbers happen to be inlined, and when not, this can
be reported as an error and we will try to figure out why not.

> 1. The CVS version of SLIME does not work.  *inferior-lisp* ends with the
> message:
> 
>   ;;; Loading "/home/neuss/CL-HOME/elisp/slime/nregex.lisp"
>   ;;; Loading "/home/neuss/CL-HOME/elisp/slime/swank-ecl.lisp"
>   Module error: Don't know how to REQUIRE SOCKETS.
>   Broken at REQUIRE.
>   SWANK-BACKEND>>

You need TCP sockets for slime to work. Configure ECL with the flag
--with-tcp and rebuild it.

> 2. I have problems with logical pathnames.  I am using:
> 
>   (defvar *femlisp-pathname*
>     (make-pathname :directory (pathname-directory *load-truename*)))
> 
>   (let ((directory (pathname-directory *femlisp-pathname*)))
>     (setf (logical-pathname-translations "FEMLISP")
>           `(("**;*.*.*"
>              ,(make-pathname :directory `(,@directory :wild-inferiors)
>                              :name :wild :type :wild :version :wild)))))
> 
>   (load "femlisp:femlisp-config.lisp")
> 
> and get an error[...] However, PROBE-FILE works:

Fixed in CVS. My latest commit to fix translation problems when :VERSION
= :WILD was a total disaster born out of the lack of sleep.

> 3. Loading my floating point performance test
> 
>   <http://sit.iwr.uni-heidelberg.de/~neuss/misc/mflop.lisp>

> with (load (compile-file "mflop.lisp")) yields
> 
> DDOT-long: 158.84 MFLOPS
> DDOT-short: 466.84 MFLOPS
> DAXPY-long: 7.84 MFLOPS
> DAXPY-short: 7.63 MFLOPS
> 
> The first two values are fine, but the other ones are much worse than GCL
> which gives:
> 
> DDOT-long: 189.04 MFLOPS
> DDOT-short: 511.31 MFLOPS
> DAXPY-long: 99.42 MFLOPS
> DAXPY-short: 447.39 MFLOPS
> 
> Why?

I am looking now into the output of the compiler. Most likely, the array
operations are not being inlined. However, I would never implement these
operations in lisp itself, but rather link against an optimized BLAS
library. I do that even when programming in C/C++.

Regards,

Juanjo

Juan Jose Garcia Ripoll <lisp@...> writes:

> You need TCP sockets for slime to work. Configure ECL with the flag
> --with-tcp and rebuild it.

Now it does not start up any more:

$ ecl

Unrecoverable error:
Lisp initialization error.

Aborted

> Fixed in CVS. My latest commit to fix translation problems when :VERSION
> = :WILD was a total disaster born out of the lack of sleep.

Are CVS changes immediately visible to others?  I probably did not get them
yet, because the problem still remains.

> I am looking now into the output of the compiler. Most likely, the array
> operations are not being inlined. However, I would never implement these
> operations in lisp itself, but rather link against an optimized BLAS
> library. I do that even when programming in C/C++.

OK, but at some points I have to do other FP operations than BLAS.  If an
implementation fails to optimize those simple loops, it will probably fail
also for the more complicated cases.  Furthermore, having those routines in
Lisp adds quite some portability, because FFI is not standardized.

> Regards,
>
> Juanjo

Thanks you,
Nicolas.

On Thu, 2005-08-04 at 19:26 +0200, Nicolas Neuss wrote:
> Juan Jose Garcia Ripoll <lisp@...> writes:
> 
> > You need TCP sockets for slime to work. Configure ECL with the flag
> > --with-tcp and rebuild it.
> 
> Now it does not start up any more:
> 
> $ ecl
> Unrecoverable error:
> Lisp initialization error.

This is strange. Configuring --with-tcp and without it should make no
real difference, as it only decides whether to build some extra module
or not (the code in the library for it is minimal). Did you remove the
_build_ directory before configuring again?
> 
> > Fixed in CVS. My latest commit to fix translation problems when :VERSION
> > = :WILD was a total disaster born out of the lack of sleep.
> 
> Are CVS changes immediately visible to others?  I probably did not get them
> yet, because the problem still remains.

There is a delay for anonymous access. However one my test machines uses
the anonymous repository and it built and tested successfully. See
Redhat 8.0 in http://ecls.sourceforge.net/status.html

> > I am looking now into the output of the compiler. Most likely, the array
> > operations are not being inlined. However, I would never implement these
> > operations in lisp itself, but rather link against an optimized BLAS
> > library. I do that even when programming in C/C++.
> 
> OK, but at some points I have to do other FP operations than BLAS.  If an
> implementation fails to optimize those simple loops, it will probably fail
> also for the more complicated cases.  Furthermore, having those routines in
> Lisp adds quite some portability, because FFI is not standardized.

UFFI is a defacto standard, which is supported by most Common-Lisp
implementations. However, even though you are right and UFFI will not be
capable of, for instance, retreiving the pointer of a single-float
array, typically the code that this adds is minimal and one can design a
good interface on top of this. I think Matlisp does this.

Incidentally, I had a look at your code and indeed an array operation is
not being inlined and thus spoils all the performance gain from unboxing
the real numbers. I will work into that.

Regards,

Juanjo

Hi,

I did see this once.  I think that not everything got rebuilt.  It
might have been happening because the ecl executable was not finding
the right libecl.so file.  No not...

It was solved by removing the ecl-0.9f subdir, reuntaring the files,
./configure, and make/make install.

cheers

bruce

On Fri, Aug 05, 2005 at 10:40:34AM +0200, Juan Jose Garcia Ripoll wrote:
> On Thu, 2005-08-04 at 19:26 +0200, Nicolas Neuss wrote:
> > Juan Jose Garcia Ripoll <lisp@...> writes:
> > 
> > > You need TCP sockets for slime to work. Configure ECL with the flag
> > > --with-tcp and rebuild it.
> > 
> > Now it does not start up any more:
> > 
> > $ ecl
> > Unrecoverable error:
> > Lisp initialization error.
> 
> This is strange. Configuring --with-tcp and without it should make no
> real difference, as it only decides whether to build some extra module
> or not (the code in the library for it is minimal). Did you remove the
> _build_ directory before configuring again?
> > 
> > > Fixed in CVS. My latest commit to fix translation problems when :VERSION
> > > = :WILD was a total disaster born out of the lack of sleep.
> > 
> > Are CVS changes immediately visible to others?  I probably did not get them
> > yet, because the problem still remains.
> 
> There is a delay for anonymous access. However one my test machines uses
> the anonymous repository and it built and tested successfully. See
> Redhat 8.0 in http://ecls.sourceforge.net/status.html
> 
> > > I am looking now into the output of the compiler. Most likely, the array
> > > operations are not being inlined. However, I would never implement these
> > > operations in lisp itself, but rather link against an optimized BLAS
> > > library. I do that even when programming in C/C++.
> > 
> > OK, but at some points I have to do other FP operations than BLAS.  If an
> > implementation fails to optimize those simple loops, it will probably fail
> > also for the more complicated cases.  Furthermore, having those routines in
> > Lisp adds quite some portability, because FFI is not standardized.
> 
> UFFI is a defacto standard, which is supported by most Common-Lisp
> implementations. However, even though you are right and UFFI will not be
> capable of, for instance, retreiving the pointer of a single-float
> array, typically the code that this adds is minimal and one can design a
> good interface on top of this. I think Matlisp does this.
> 
> Incidentally, I had a look at your code and indeed an array operation is
> not being inlined and thus spoils all the performance gain from unboxing
> the real numbers. I will work into that.
> 
> Regards,
> 
> Juanjo
> 
> 
> 
> -------------------------------------------------------
> SF.Net email is Sponsored by the Better Software Conference & EXPO
> September 19-22, 2005 * San Francisco, CA * Development Lifecycle Practices
> Agile & Plan-Driven Development * Managing Projects & Teams * Testing & QA
> Security * Process Improvement & Measurement * http://www.sqe.com/bsce5sf
> _______________________________________________
> Ecls-list mailing list
> Ecls-list@...
> https://lists.sourceforge.net/lists/listinfo/ecls-list

-- 
edoneel@...
SDF Public Access UNIX System - http://sdf.lonestar.org

Juan Jose Garcia Ripoll <lisp@...> writes:

> This is strange. Configuring --with-tcp and without it should make no
> real difference, as it only decides whether to build some extra module
> or not (the code in the library for it is minimal). Did you remove the
> _build_ directory before configuring again?

Removing the build directory helped.  Seems to work fine now.  Thanks.

> the anonymous repository and it built and tested successfully. See
> Redhat 8.0 in http://ecls.sourceforge.net/status.html

Indeed, I was too fast checking out and have got the patch only now.  Do
you know how large this anonymous cvs delay is?

Now, I am stuck when performing "(asdf:operate 'asdf::load-op 'femlisp)",
which may indicate a bug in your ASDF port.  I get

The variable *COMPILE-PRINT* is unbound.
[Condition of type UNBOUND-VARIABLE]

Loading single files apparently works.  I will try to go through that.
One thing that showed up is the following LOOP construct:

(let ((list (list 1 2 3))
      (result ()))
   (loop for i = (pop list) while i
         for k = (* i i) do (push k result)
         finally (return result)))

Is this construct really non-ANSI?  It worked in Allegro, CMUCL, SBCL.

> UFFI is a defacto standard, which is supported by most Common-Lisp
> implementations. However, even though you are right and UFFI will not be
> capable of, for instance, retreiving the pointer of a single-float array,
> typically the code that this adds is minimal and one can design a good
> interface on top of this. I think Matlisp does this.

Hmm, I wanted to ask if Matlisp has been ported to ECL, but Michael has
already answered this.  Indeed, Femlisp once depended on Matlisp, but I
found that really inconvenient.  If Matlisp was more widely used and would
provide a more complete interface to BLAS/LAPACK/..., the situation would
probably be different.

> Incidentally, I had a look at your code and indeed an array operation is
> not being inlined and thus spoils all the performance gain from unboxing
> the real numbers. I will work into that.
>
> Regards,
>
> Juanjo

Thanks again,
Nicolas.

On Fri, 2005-08-05 at 16:17 +0200, Nicolas Neuss wrote:
> Juan Jose Garcia Ripoll <lisp@...> writes:
> > the anonymous repository and it built and tested successfully. See
> > Redhat 8.0 in http://ecls.sourceforge.net/status.html
> 
> Indeed, I was too fast checking out and have got the patch only now.  Do
> you know how large this anonymous cvs delay is?

Well, it was supposed to be zero after the CVS infrastructure was moved
to different servers. But in practice seems to be around 5 hours.

> Now, I am stuck when performing "(asdf:operate 'asdf::load-op 'femlisp)",
> which may indicate a bug in your ASDF port.  I get
> 
> The variable *COMPILE-PRINT* is unbound.
> [Condition of type UNBOUND-VARIABLE]
> 
> Loading single files apparently works.  I will try to go through that.

I would be very grateful if you could track this bug. I am rather busy
and, though I can probably solve problems once spotted and well
delimited, I have not much time for debugging.

> One thing that showed up is the following LOOP construct:
> 
> (let ((list (list 1 2 3))
>       (result ()))
>    (loop for i = (pop list) while i
>          for k = (* i i) do (push k result)
>          finally (return result)))
> 
> Is this construct really non-ANSI?  It worked in Allegro, CMUCL, SBCL.

It is not standard. If you read the Hyperspec you will find the syntax
	loop [name-clause] {variable-clause}* {main-clause}* => result*
WHILE belongs to main-clause while FOR belongs to variable-clause.
The fact that something works on an implementation does not mean it is
correct :-)

Regards

Juanjo

On Fri, 05 Aug 2005 16:17:33 +0200
Nicolas Neuss <Nicolas.Neuss@...> wrote:

> Loading single files apparently works.  I will try to go through that.
> One thing that showed up is the following LOOP construct:
> 
> (let ((list (list 1 2 3))
>       (result ()))
>    (loop for i = (pop list) while i
>          for k = (* i i) do (push k result)
>          finally (return result)))

Is there any reason why this is written so complicated (besides mixing
iterating forms with loop end tests)? Looks like

(for i in list
     collect (* i i))

to me. Anyway, you might be better of using iterate. It solved a lot of
my loop problems.
 
Regards,
-- 
Julian Stecklina

LISP has survived for 21 years because it is an approximate local
optimum in the space of programming languages. - John McCarthy (1980)

Julian Stecklina <der_julian@...> writes:

> Is there any reason why this is written so complicated (besides mixing
> iterating forms with loop end tests)?

Of course the real thing involved some functions not known to a general
public.  I thought that people would prefer a version they could test
themselves without porting Femlisp to ECL.

Nicolas.

Juan Jose Garcia Ripoll <lisp@...> writes:

> Well, it was supposed to be zero after the CVS infrastructure was moved
> to different servers. But in practice seems to be around 5 hours.

Oh, that's a very long gap.

> I would be very grateful if you could track this bug. I am rather busy
> and, though I can probably solve problems once spotted and well
> delimited, I have not much time for debugging.

Let's start with this.  Put the following in a file and do (load
(compile-file ...)), [C-k] from slime.  Results for me in an error that the
device "femlisp:" is not known.

---------------------------------------------------------------------------
(in-package :cl-user)
(defvar *femlisp-pathname* #p"/home/neuss/CL-HOME/femlisp/")
(let ((directory (pathname-directory *femlisp-pathname*)))
  (setf (logical-pathname-translations "FEMLISP")
        `(("**;*.*.*"
           ,(make-pathname :directory `(,@directory :wild-inferiors)
                           :name :wild :type :wild :version :wild)))))
(require 'asdf)
(pushnew (probe-file #p"femlisp:") asdf::*central-registry* :test #'equalp)
---------------------------------------------------------------------------

Another bug (I think):

  (defmethod foo (x) (defmethod bar ((f (eql x)))))

does not work.

Yours, Nicolas.

Nicolas Neuss <Nicolas.Neuss@...> writes:

> Julian Stecklina <der_julian@...> writes:
>
>> Is there any reason why this is written so complicated (besides mixing
>> iterating forms with loop end tests)?
>
> Of course the real thing involved some functions not known to a general
> public.  I thought that people would prefer a version they could test
> themselves without porting Femlisp to ECL.

Sorry, I looked again and the answer is more interesting.  The code came
from my testing setup and was

(loop for fsym = (pop *testing*) while fsym
   for result =
        (catch 'trap
          (handler-bind ((error #'(lambda (condition) (throw 'trap condition))))
            (format t "~&~%***** Testing ~A *****~%~%" fsym)
            (funcall fsym)
            nil))
   when result do (push (cons fsym result) *failed*)
   finally (return *failed*))

I rewrote it now as (loop ... (let ((result ...)) ...) ...) which costs one
indentation level.  But I guess that I should indeed use LOOP less often.
The ordering of execution of initialization, stepping, and termination
clauses is quite unintuitive.

Nicolas.

On Fri, 05 Aug 2005 20:51:51 +0200, <Nicolas.Neuss@...> wrote:
>
> But I guess that I should indeed use LOOP less often.
> The ordering of execution of initialization, stepping, and termination
> clauses is quite unintuitive.

I think we had this discussion already, but Jon L White's LOOP chapter
in Steele is rather clear about execution order in my opinion.


-- 
For when I found the holy lisp
So pure the curves that are parens
With a report merely one short text
Of beauty's simplicity    --r2q2

On Fri, 2005-08-05 at 20:40 +0200, Nicolas Neuss wrote:
> Let's start with this.  Put the following in a file and do (load
> (compile-file ...)), [C-k] from slime.  Results for me in an error that the
> device "femlisp:" is not known. [...]

Please do not take me as a lazy guy, but could you rather supply an http
address where I can download the package that causes the error? I mean
something I can just unpack and load.

> Another bug (I think):
> 
>   (defmethod foo (x) (defmethod bar ((f (eql x)))))
> 
> does not work.

Fixed in CVS.

Juanjo

On Fri, 2005-08-05 at 20:40 +0200, Nicolas Neuss wrote: 
> Let's start with this.  Put the following in a file and do (load
> (compile-file ...)), [C-k] from slime.  Results for me in an error that the
> device "femlisp:" is not known.
> 
> ---------------------------------------------------------------------------
> (in-package :cl-user)
> (defvar *femlisp-pathname* #p"/home/neuss/CL-HOME/femlisp/")
> (let ((directory (pathname-directory *femlisp-pathname*)))
>   (setf (logical-pathname-translations "FEMLISP")
>         `(("**;*.*.*"
>            ,(make-pathname :directory `(,@directory :wild-inferiors)
>                            :name :wild :type :wild :version :wild)))))
> (require 'asdf)
> (pushnew (probe-file #p"femlisp:") asdf::*central-registry* :test #'equalp)
> ---------------------------------------------------------------------------

No real problem here :-/

> (in-package :cl-user)
(defvar *femlisp-pathname* #p"/home/jlr/src/lisp/femlisp/")
(let ((directory (pathname-directory *femlisp-pathname*)))
  (setf (logical-pathname-translations "FEMLISP")
        `(("**;*.*.*"
           ,(make-pathname :directory `(,@directory :wild-inferiors)
                           :name :wild :type :wild :version :wild)))))
(require 'asdf)
(pushnew (probe-file #p"femlisp:") asdf::*central-registry* :test
#'equalp)

#<"COMMON-LISP-USER" package>
> *FEMLISP-PATHNAME*
> ((#P"FEMLISP:**;*.*.*" #P"/home/jlr/src/lisp/femlisp/**/*.*"))
> ;;; Loading #P"/home/jlr/lib/ecl/asdf.fas"
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: PRINT-OBJECT is being redefined.
;;; Warning: SHARED-INITIALIZE is being redefined.
("ASDF")
> (#P"/home/jlr/src/lisp/femlisp/" *DEFAULT-PATHNAME-DEFAULTS*)

Juanjo

Juan Jose Garcia Ripoll <lisp@...> writes:

> On Fri, 2005-08-05 at 20:40 +0200, Nicolas Neuss wrote:
>> Let's start with this.  Put the following in a file and do (load
>> (compile-file ...)), [C-k] from slime.  Results for me in an error that the
>> device "femlisp:" is not known. [...]
>
> Please do not take me as a lazy guy, but could you rather supply an http
> address where I can download the package that causes the error? I mean
> something I can just unpack and load.

Femlisp is available at http://www.femlisp.org, but my corrections for ECL have
not yet been incorporated.  Hmm, maybe the above was a false alarm, because
at the moment Femlisp starts up just fine from SLIME/ECL (but compilation
breaks later on).  Therefore, I would suggest to postpone this issue.

If you agree I would suggest that we continue in the following way: I will
report always the next problem (ECL or Femlisp bug), trying to distill out
a small test case.  Then you can comment on it, showing that I deviate from
the standard or providing a fix or suggesting a work around.  I think that
the work is best partitioned in this way.

At the moment, this next problem lies with constructs like the following:

(DEFMETHOD GEMM (ALPHA X Y BETA Z)
  (MACROLET ((WITH-BLAS-DATA (MATRICES &REST BODY)
               `(MACROLET ()
                   ,@BODY)))
    (WITH-BLAS-DATA (Z Y X)
      (DECLARE (OPTIMIZE SPEED))
      Z)))

--> The function WALKER::WALK-UNEXPECTED-DECLARE is undefined.

As much as I understand the spec, declare should be allowed in this
position.

Yours, Nicolas.

P.S.: These snippets might be nice contributions to Paul Dietz' ANSI test
suite.  Paul, are you reading this?

On Wed, 2005-08-10 at 23:21 +0200, Nicolas Neuss wrote:
> At the moment, this next problem lies with constructs like the following:
> 
> (DEFMETHOD GEMM (ALPHA X Y BETA Z)
>   (MACROLET ((WITH-BLAS-DATA (MATRICES &REST BODY)
>                `(MACROLET ()
>                    ,@BODY)))
>     (WITH-BLAS-DATA (Z Y X)
>       (DECLARE (OPTIMIZE SPEED))
>       Z)))

Seems like a problem with ECL's code walker. Unfortunately, this library
is required currently by CLOS. It is one of my priorities to get rid of
this dependency, but after a couple of days looking at CALL-NEXT-METHOD
it seems it is going to be a hard job.

I'll look into fixing the code walker, instead.

Sorry for the delay,

Juanjo

Juan Jose Garcia Ripoll <lisp@...> writes:

> Seems like a problem with ECL's code walker. Unfortunately, this library
> is required currently by CLOS. It is one of my priorities to get rid of
> this dependency, but after a couple of days looking at CALL-NEXT-METHOD
> it seems it is going to be a hard job.
>
> I'll look into fixing the code walker, instead.
>
> Sorry for the delay,
>
> Juanjo

Thanks for looking into it.  In the meantime I have changed my code
(locally, not yet in CVS) and compilation does not stop anymore because of
this bug.  Unfortunately, ECL breaks later when compiling a rather
complicated construct for which I did not yet succeed with constructing a
small sample.

Yours, Nicolas.

Nicolas Neuss <Nicolas.Neuss@...> writes:

> In the meantime I have changed my code (locally, not yet in CVS) and
> compilation does not stop anymore because of this bug.  Unfortunately,
> ECL breaks later when compiling a rather complicated construct for which
> I did not yet succeed with constructing a small sample.

Here is the reduced sample

(defmethod gemm-nn! ((alpha number) x y (beta number) z)
  (macrolet ((matrix-loop (bindings &rest body)
	       nil))
    (matrix-loop ((z :col-index))
		 nil)
    z))

It throws the following error:

Can't get template for (Z :COL-INDEX)
   [Condition of type SIMPLE-ERROR]

Yours, Nicolas.

On Mon, 2005-09-05 at 17:06 +0200, Nicolas Neuss wrote:
> Nicolas Neuss <Nicolas.Neuss@...> writes:
> 
> > In the meantime I have changed my code (locally, not yet in CVS) and
> > compilation does not stop anymore because of this bug.  Unfortunately,
> > ECL breaks later when compiling a rather complicated construct for which
> > I did not yet succeed with constructing a small sample.
> 
> Here is the reduced sample
> 
> (defmethod gemm-nn! ((alpha number) x y (beta number) z)
>   (macrolet ((matrix-loop (bindings &rest body)
> 	       nil))
>     (matrix-loop ((z :col-index))
> 		 nil)
>     z))

Hi,

It has taken me quite some time, but finally I got rid of the code
walker in ECL. There should be no significant penalty in any kind of
CLOS code, except for slot access, which is now faster when using
accessors than when using SLOT-VALUE. The reason is that I had to remove
some optimizations that were arguably insecure when combined with the
possibility of redefining classes.

BTW, I just arrived from several trips abroad including holidays and
conferences and it will take me some time to catch up with the errors
and problems reported in the mailing list.

Regards,

Juanjo

On Mon, 2005-09-19 at 11:35 +0200, Juan Jose Garcia Ripoll wrote:
> It has taken me quite some time, but finally I got rid of the code
> walker in ECL.

As a side effect, this means stripping 60k of code and data from the
core library. It would be nice to come up with similar optimizations on
other fields (pprint, format, loop, etc) :-)

Juanjo

Hello Juan Jose,

thank you very much for this patch.  This part of Femlisp does work now
with ECL (even the tests of this part are passing), and I think the hardest
part is done.  Nevertheless, there is still a problems: ECL yields a core
dump when compiling the file "femlisp/src/algebra/sparse.lisp".  I do not
know why (loading the definitions separately seems to work), and I wanted
to ask you to look at this directly.  Maybe you have some means to print
out some information which helps you to find out where it goes wrong.

It should be straightforward to compile Femlisp:

(1) Download:

  cvs -z3 -d:ext:anoncvs@... co femlisp

(2) Configure:

  cd femlisp/
  make configure  # simply enter the full path to ecl at the first question


(3) Compile

  make femlisp

Alternatively to (2) and (3), loading the file "femlisp/start.lisp" into
ecl should also compile Femlisp.

Thanks again,

Nicolas.

Juan Jose Garcia Ripoll <lisp@...> writes:

> It has taken me quite some time, but finally I got rid of the code
> walker in ECL. There should be no significant penalty in any kind of
> CLOS code, except for slot access, which is now faster when using
> accessors than when using SLOT-VALUE. The reason is that I had to remove
> some optimizations that were arguably insecure when combined with the
> possibility of redefining classes.
>
> BTW, I just arrived from several trips abroad including holidays and
> conferences and it will take me some time to catch up with the errors
> and problems reported in the mailing list.
>
> Regards,
>
> Juanjo

On Wed, 2005-09-21 at 14:44 +0200, Nicolas Neuss wrote: 
> [...] This part of Femlisp does work now
> with ECL (even the tests of this part are passing), and I think the hardest
> part is done.  Nevertheless, there is still a problems: ECL yields a core
> dump when compiling the file "femlisp/src/algebra/sparse.lisp". 

I have tried (not very hard because of time constraints) to compile
femlisp for the last weeks, but I always end up with a core dump, but
due to the fact that ECL ate all available memory. I suspect there must
be either some problem in the compiler, or ECL cannot handle so many
FASL files (are they loaded during compilation?)

Alternatively I change start.lisp to work with the new ASDF automatized
joint FASL building (i.e. combining all compiled files into a single
shared library that ECL can easily handle). Basically

Index: start.lisp
===================================================================
RCS file: /cvsroot/femlisp/femlisp/start.lisp,v
retrieving revision 1.15
diff -u -r1.15 start.lisp
--- start.lisp	13 Oct 2005 16:11:09 -0000	1.15
+++ start.lisp	24 Oct 2005 13:35:06 -0000
@@ -151,7 +151,10 @@
 ;;; we want to work generally with double float numbers
 (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float)
 
+#-ecl
 (asdf:operate 'asdf::load-op :femlisp-src)
+#+ecl
+(asdf::make-build :femlisp-src :type :fasl)
 
 (pushnew :femlisp *features*)

But here I found problems with LOOP. Constructs like the ones showed
below are wrong because you cannot intermingle WHILE and FOR statements
in a loop. It is not ANSI, even though some lisps seem to swallow it.

For me it is difficult to correct the statements because I do not know
what evaluation order you expect in that code. Could you please add the
previous tweak, maybe rewrite the LOOP statements and give ECL a try?

Regards

Juanjo

(defun adapt-fill-pointer (table)
  (loop while (plusp (fill-pointer table))
	for dll = (aref table (1- (fill-pointer table)))
	while (or (null dll) (dll-empty-p dll))
	do (decf (fill-pointer table))))

(defmethod extend-triangulation (mesh (dim (eql 0)) &rest args)
  (let* ((domain (domain mesh))
	 (patches (cells-of-dim domain 0)))
    (loop
     while patches for patch = (first patches) do
     ^^^^^         ^^^

Juan Jose Garcia Ripoll <lisp@...> writes:

> On Wed, 2005-09-21 at 14:44 +0200, Nicolas Neuss wrote: 
>> [...] This part of Femlisp does work now
>> with ECL (even the tests of this part are passing), and I think the hardest
>> part is done.  Nevertheless, there is still a problems: ECL yields a core
>> dump when compiling the file "femlisp/src/algebra/sparse.lisp". 
>
> I have tried (not very hard because of time constraints) to compile
> femlisp for the last weeks, but I always end up with a core dump, but
> due to the fact that ECL ate all available memory. I suspect there must
> be either some problem in the compiler, or ECL cannot handle so many
> FASL files (are they loaded during compilation?)

I think they are compiled and loaded in chunks - whatever ASDF does...

> Alternatively I change start.lisp to work with the new ASDF automatized
> joint FASL building (i.e. combining all compiled files into a single
> shared library that ECL can easily handle). Basically
>
> Index: start.lisp
> ===================================================================
> RCS file: /cvsroot/femlisp/femlisp/start.lisp,v
> retrieving revision 1.15
> diff -u -r1.15 start.lisp
> --- start.lisp	13 Oct 2005 16:11:09 -0000	1.15
> +++ start.lisp	24 Oct 2005 13:35:06 -0000
> @@ -151,7 +151,10 @@
>  ;;; we want to work generally with double float numbers
>  (setq *READ-DEFAULT-FLOAT-FORMAT* 'double-float)
>  
> +#-ecl
>  (asdf:operate 'asdf::load-op :femlisp-src)
> +#+ecl
> +(asdf::make-build :femlisp-src :type :fasl)
>  
>  (pushnew :femlisp *features*)

OK, I'll incorporate this.

> But here I found problems with LOOP. Constructs like the ones showed
> below are wrong because you cannot intermingle WHILE and FOR statements
> in a loop. It is not ANSI, even though some lisps seem to swallow it.
>
> For me it is difficult to correct the statements because I do not know
> what evaluation order you expect in that code. Could you please add the
> previous tweak, maybe rewrite the LOOP statements and give ECL a try?

I'll try to do that in the next days.

Thank you,

Nicolas.

P.S.: In the meantime I have succeeded in getting GCL to compile and load
Femlisp without errors which should help the ECL port.  However, up to now
the Femlisp test suite does not run through, partly because there are
operations for which GCL needs much more resources than CMUCL/SBCL/Allegro.
Therefore, I fear that there is some work to be done (including profiling
and special optimizations) until either the ECL or GCL port reaches a
useful state.

Juan Jose Garcia Ripoll <lisp@...> writes:

> ...

OK, I followed your suggestions. Then I updated ECL from CVS and tried to
compile it.  It got stuck at the following point:

dpp: /home/neuss/ecls/src/c/ffi.d -> ffi.c

Unknown symbol: mp::+load-compile-lock+

Unknown symbol: mp::+load-compile-lock+
gcc -c -I. -I/home/neuss/ecls/src/c -I/home/neuss/ecls/src/h -I../h  -I../includ
e -g -O2 -fPIC -fstrict-aliasing -Dlinux  -o ffi.o ffi.c
if test -f ../CROSS-DPP ; then ../CROSS-DPP /home/neuss/ecls/src/c/alloc_2.d all
oc_2.c ; else ./dpp /home/neuss/ecls/src/c/alloc_2.d alloc_2.c ; fi
dpp: /home/neuss/ecls/src/c/alloc_2.d -> alloc_2.c
gcc -c -I. -I/home/neuss/ecls/src/c -I/home/neuss/ecls/src/h -I../h  -I../includ
e -g -O2 -fPIC -fstrict-aliasing -Dlinux  -o alloc_2.o alloc_2.c
/home/neuss/ecls/src/c/alloc_2.d: In function `stacks_scanner':
/home/neuss/ecls/src/c/alloc_2.d:294: warning: passing arg 1 of `GC_set_mark_bit
' from incompatible pointer type
/home/neuss/ecls/src/c/alloc_2.d:297: warning: passing arg 1 of `GC_set_mark_bit
' from incompatible pointer type
make[2]: *** No rule to make target `ffi_x86.c', needed by `ffi_x86.o'.  Stop.
rm array.c symbol.c num_rand.c num_sfun.c structure.c cmpaux.c list.c unixsys.c 
num_arith.c load.c string.c stacks.c hash.c main.c predicate.c num_log.c unixint
.c instance.c compiler.c character.c multival.c gfun.c sequence.c ffi.c unixfsys
.c reference.c time.c print.c backq.c format.c typespec.c num_co.c num_pred.c ma
cros.c alloc_2.c read.c error.c num_comp.c number.c pathname.c file.c assignment
.c cfun.c disassembler.c interpreter.c mapfun.c package.c eval.c
make[2]: Leaving directory `/home/neuss/ecls/build/c'
make[1]: *** [libeclmin.a] Error 2
make[1]: Leaving directory `/home/neuss/ecls/build'
make: *** [all] Error 2

Yours, Nicolas.

On Tue, 2005-10-25 at 16:03 +0200, Nicolas Neuss wrote:
> OK, I followed your suggestions. Then I updated ECL from CVS and tried to
> compile it.  It got stuck at the following point: [...]
> make[2]: *** No rule to make target `ffi_x86.c', needed by `ffi_x86.o'.  Stop.

You have to update with

	cvs update -C -P -d

so that the new directory ecls/src/c/arch and the associated files are
also downloaded

Juanjo

Juan Jose Garcia Ripoll <lisp@...> writes:

> You have to update with
>
> 	cvs update -C -P -d
>
> so that the new directory ecls/src/c/arch and the associated files are
> also downloaded

OK, thanks, I should have thought of that.  Now it compiled[1] and it looks
as if many things are working.  Ufortunately, I think that a lot of
optimization work is necessary.  At the moment, ECL is more than 20 times
slower than any of Allegro/CMUCL/SBCL, probably for a large part due to a
suboptimal compilation of type-adapted BLAS methods.

When I find time I will examine this further.

Anyway, many thanks for your help,

Nicolas.

[1] The problem with compiling and loading sparse.lisp was still there for
me.  It disappeared when I omitted a CALL-NEXT-METHOD as follows:

(defmethod mzerop ((smat <sparse-matrix>) &optional (threshold 0.0))
  (declare (ignore threshold))
  (or (zerop (nr-nonempty-rows smat))
      #-ecl(call-next-method)))

On Wed, 2005-10-26 at 19:39 +0200, Nicolas Neuss wrote:
> [1] The problem with compiling and loading sparse.lisp was still there for
> me.  It disappeared when I omitted a CALL-NEXT-METHOD as follows:

There are issues with CALL-NEXT-METHOD, as the ANSI test suite showed us
long ago. Unfortunately, though it is I would say a critical bug, this
deals with the MOP and that means it will have to wait until I find two
free days on a row. :-)

Anyway, thanks for testing ECL with femlisp. I think _that_ is the type
of programs we should be looking at to see what needs to be done or
improved in ECL.

BTW, I saw you use PBS on some of the scripts. Is femlisp a parallelized
program itself?

Regards,

Juanjo

Juan Jose Garcia Ripoll <lisp@...> writes:

> Anyway, thanks for testing ECL with femlisp. I think _that_ is the type
> of programs we should be looking at to see what needs to be done or
> improved in ECL.

I tend to agree.  As much as I see, ECL and GCL both have some potential
for numerical applications due to their possibly better integration with
C/C++ libraries.  If the performance problems can be solved...

> BTW, I saw you use PBS on some of the scripts. Is femlisp a parallelized
> program itself?

Not yet, but I would like to make it so.  I have run it on a 32 node
cluster (each node having 2 AMD processors with 2GB RAM), but unfortunately
only as a serial application.  Using two processors for some
computationally intensive parts should be easily possible.  Using all nodes
would need a reasonable MPI interface.

> Regards,
>
> Juanjo


Yours, Nicolas

On Oct 27, 2005, at 5:27 AM, Nicolas Neuss wrote:

>
>> BTW, I saw you use PBS on some of the scripts. Is femlisp a 
>> parallelized
>> program itself?
>
> Not yet, but I would like to make it so.  I have run it on a 32 node
> cluster (each node having 2 AMD processors with 2GB RAM), but 
> unfortunately
> only as a serial application.  Using two processors for some
> computationally intensive parts should be easily possible.  Using all 
> nodes
> would need a reasonable MPI interface.

I am getting old,  but wasn't there a PVM interface in the old EcoLisp?

Cheers



--
Marco Antoniotti					http://bioinformatics.nyu.edu/~marcoxa
NYU Courant Bioinformatics Group		tel. +1 - 212 - 998 3488
715 Broadway 10th FL				fax. +1 - 212 - 998 3484
New York, NY, 10003, U.S.A.

On Thu, 2005-10-27 at 11:50 -0400, Marco Antoniotti wrote:
> I am getting old,  but wasn't there a PVM interface in the old EcoLisp?

Yes, it was in the contrib/ directory and can be retrieved from CVS with
the appropiate version number. However it does not work with the current
ECL. In any case, most clusters I know these days ask for and provide
MPI, not PVM.

Juanjo

--=20
Max-Planck-Institut f=C3=BCr Quantenoptik
Hans-Kopfermann-Str. 1, Garching, D-85748, Germany
Phone: +49 89 32905 345   Fax: +49 89 32905 336
http://www.mpq.mpg.de/Theorygroup/CIRAC/

On Thu, 27 Oct 2005 11:27:21 +0200
Nicolas Neuss <Nicolas.Neuss@...> wrote:

> Not yet, but I would like to make it so.  I have run it on a 32 node
> cluster (each node having 2 AMD processors with 2GB RAM), but
> unfortunately only as a serial application.  Using two processors for
> some computationally intensive parts should be easily possible.
> Using all nodes would need a reasonable MPI interface.

It should not be complicated to whip up a small MPI interface for
testing. But I am quite puzzled how to implement the fancy datatype
stuff of MPI, e.g. sending a column of a 2D double array and
receive it as row. (Is this possible with MPI?) But having the datatype
functions wrapped using convenience macros would simplify a lot of
things. Just dreaming... Seems like I have to write my homework tasks
in C and Fortran (yak!) anyway. ;)

Regards,
-- 
Julian Stecklina

When someone says "I want a programming language in which I
need only say what I wish done," give him a lollipop. - Alan Perlis

Julian Stecklina <der_julian@...> writes:

> It should not be complicated to whip up a small MPI interface for
> testing. [...]

There is also ParGCL which could possibly be ported.  Disclaimer: I did not
yet look at it.

Nicolas.