clisp-devel — closed list for developers and testers

Re: GSoC status report

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-17 03:31:58 +0200]:
>
>>(integer-length most-positive-fixnum) should return 48.
>
> It does.

Sehr gut!

> I also found why I get those ffi errors:
> "We" define HAVE_LONG_LONG_INT. It gets undefined in (at least my version of)
> avcall.h, which contains this relatively at the top:
>
> /* AC_TYPE_LONG_LONG */
> /* Define if your compiler supports the 'long long' type. */
> #undef HAVE_LONG_LONG_INT
>
> /* End of definitions adjusted by `configure'. */

Okay, so this is a bug in libffcall configure.
Not your problem.
Please report it to the libffcall people and either edit avcall.h
directly or redefined HAVE_LONG_LONG_INT in foreign.d after including
avcall.h.

> HEAD of libffcall defines HAVE_LONG_LONG_INT to 1 as it should on my
> system. (I was using 2009-05-27 previously.)

Cool, problem solved.
Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://www.memritv.org http://ffii.org
http://memri.org http://honestreporting.com http://mideasttruth.com
Any fool can squander everything once. Doing it more than once requires
genius.

Re: case sensitivity of provide, require and module-provide-asdf

[Sam S. <sd...@gn...>]

> * Pascal J. Bourguignon <cwo@vasbezngvzntb.pbz> [2016-05-15 23:05:38 +0200]:
>
> cl-user> (eq #P"/tmp/Foo" #P"/tmp/foo")
> nil

you meant equal, not eq, right?

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://jihadwatch.org http://honestreporting.com
http://think-israel.org http://iris.org.il http://www.memritv.org
We have preferences. You have biases. They have prejudices.

Re: case sensitivity of provide, require and module-provide-asdf

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-15 21:40:41 +0200]:
>
> Given the ignorance of case prevalent in Common Lisp,

the only places where this happens are:
1. reader is case-converting by default
2. logical pathnames are case-insensitive

otherwise case matters.

> What are your thoughts on this?

This is far out of scope for the project.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://mideasttruth.com http://think-israel.org
http://jihadwatch.org http://ffii.org http://thereligionofpeace.com
A PC without Windows is like ice cream without ketchup.

Re: GSoC status report

[Daniel J. <dan...@gm...>]

Another update:

I've been concentrating on the module system as well as the modules since
the last update.  In order to get more familiar with it, I wrote a module
(using FFI) for libmagic. It's a bit "unlispy", though, providing only raw
access to the C functions and macros. (Passing keywords instead of or'ing
flags would probably be more idiomatic.)

Regarding gnulib and modules: One issue is about linking against gnulib
both for the main CLISP binary as well as for modules (like bindings/glibc).
In https://sourceforge.net/p/clisp/bugs/634/ you mention that the increased
size is not an issue on linux but possibly on other targets. Do you know
which targets?

I fear that there's not really a much better solution, though. One could
try to isolate a set of gnulib modules that are "guaranteed" to be (including
all unused functions, thus completely) linked with the CLISP binary, so that
all CLISP modules don't need to link against these parts of gnulib. This
might be fragile to maintain and keep in synch, though.

I also remember a post on the gnulib list (I think from Bruno) on building
part of gnulib as a shared library, but cannot find atm. (I think it was
MinGW related, too)

I've been also looking at generating the FFI forms (def-call-out and
friends) automatically (using libclang in order to parse header files), but
this is a bit out of scope.

Re: GSoC status report

[Daniel J. <dan...@gm...>]

Just a quick update:

Sam wrote:
> does GENERATIONAL_GC get enabled?

Yes, at least according to clisp --version.

Sam wrote:
>> Jörg wrote (regarding "64 bit support"):
>>> I'm confused. If on amd64, you're building a 64bit version, aren't you?
>>> So what doesn't get configured and what builds and passes tests?
>>
>> (on linux)
>> I'm on amd64, and am building a 64bit version. Both clisp and lisp.run are
>> 64bit elf executables. There's :WORD-SIZE=64 in *features*.
>
>(integer-length most-positive-fixnum) should return 48.

It does. I also found why I get those ffi errors:

"We" define HAVE_LONG_LONG_INT. It gets undefined in (at least my version of)
avcall.h, which contains this relatively at the top:

/* AC_TYPE_LONG_LONG */
/* Define if your compiler supports the 'long long' type. */
#undef HAVE_LONG_LONG_INT

/* End of definitions adjusted by `configure'. */

Since foreign.d includes avcall.h after including lispbibl.c, the ffi code
doesn't know that there's actually long long support.

Some first testing suggests that this is a bug in libffcall (I also suspected
my distribution, but am now pretty sure I ruled that out). HEAD of libffcall
defines HAVE_LONG_LONG_INT to 1 as it should on my system. (I was using
2009-05-27 previously.)


Regarding my issues with paths when building on windows: This is my fault,
I read configure and was assuming I could do an out-of-source build (e.g.
mkdir build; cd build; ../clisp/configure --srcdir=../clisp). I guess this
was a false assumption :)

I'm now doing as you suggested: passing a dir arg to the top level configure.

Re: case sensitivity of provide, require and module-provide-asdf

[Pascal J. B. <pj...@in...>]

Daniel Jour <dan...@gm...> writes:

> While updating the asdf module to 3.1.7, I found that our provide and
> require functions are actually case sensitive. Given the ignorance of case
> prevalent in Common Lisp,


Oh no!  Common Lisp is VERY MUCH Case Sensitive!


cl-user> (eq '|foo| '|Foo|)
nil
cl-user> (make-package "Foo")
#<Package "Foo">
cl-user> (make-package "foo")
#<Package "foo">
cl-user> (eq (find-package "foo") (find-package "Foo"))
nil
cl-user> (eq #P"/tmp/Foo" #P"/tmp/foo")
nil
cl-user> (defparameter *h* (make-hash-table :test 'equal))
*h*
cl-user> (setf (gethash "foo" *h*) 42)
42
cl-user>  (gethash "Foo" *h*)
nil
nil
cl-user>  (gethash "foo" *h*)
42
t
cl-user> 

There is no part of CL that is not very case sensitive.


> I was surprised that the HyperSpec doesn't
> contain any notes about case sensitivity for provide and require, thus I'd
> assume that we're conforming to ANSI here, right?

Given that ultimately the module name should be mapped to a file name,
it would be dumb to specify a specific case sensitivity for module
names.  On the other given that the most common file systems are case
sensitives, it would also be dumb to expect case insensitivity here.


> All of this originated from the surprise when running CLISP with the new
> asdf, because asdf now provides both "ASDF" and "asdf" (as well as "UIOP"
> and "uiop"). Excerpt from asdf.lisp:
>
> ;; Provide both lowercase and uppercase, to satisfy more people,
> especially LispWorks users.
> (provide "asdf") (provide "ASDF")
>
> Consequently, *modules* now also contains both the lowercase and the
> uppercase names.
>
> So far so good, where it really gets surprising is in the registered module
> provider function asdf/operate::module-provide-asdf, because that function
> (part of asdf.lisp) just downcases the module name right away.

As mentionned above, there's a preference for unix file systems, that
are case sensitive, and where the prefered case for file names is
lowercase.  Hence the string-downcase.


> Thus, (require "foo"), (require "foO") and (require "FOO") will each try to
> load (via asdf) the module "foo". *modules* will only contain what the module
> actually provides, though.
>
> This feels very inconsistent to me.

Indeed.


> Some approaches:
>
> * Leave as it is.

Not good, IMO.

> * Make provide and require case insensitive.

Good, IMO.


> * Let asdf only provide either of "ASDF" or "asdf". Though this is more of a
> cosmetic fix.
> * Ask asdf about it.
>
> What are your thoughts on this?


There's also basically the problem of the default mapping by an
implementation of logical pathname components to physical pathname
components.  If the situation was clear there, then it would be quite
simple to specify a direct mapping from the module name as a component
of a pathname to a module, and therefore adopting the case sensitivity
of the underlying file system.


Usual logical pathnames components are specified to be one of:

  word---one or more uppercase letters, digits, and hyphens.

  wildcard-word---one or more asterisks, uppercase letters, digits, and
                  hyphens, including at least one asterisk, with no two

So, upper case.  However, it is allowed to use lower case letters in
logical pathname, being understood that string-upcase is taken.

Now, mapping a logical pathname to a physical pathname is specified to
use an implementation dependant mapping, but using the "native" file
system conventions.

Therefore it would seem logical that the default translation of logical
pathnames to physical pathname on a unix (POSIX) platform, would involve
a string-downcase.  Unfortunately, not all implementations agree on
this, hence the need in libraries and user code to either:

- use explicit logical pathname translations, (this is inconvenient,
  since this would mean entering an translation entry for each pathname
  into the logical host translations).

- allow the implementation use uppercase physical pathnames, while
  expecting a different mapping with different implementations, and
  therefore incompatibility between different implementations on unix
  (POSIX) platforms.

- not use logical pathnames and the implementation dependant
  translations.  Basically, this choices is to use only physical
  pathnames, and therefore let the user code perform the mapping from
  whatever name using whatever convention to the file system
  conventions.

or, my favorite:

- specify that the default mapping of logical pathnames to unix (or
  POSIX) physical pathnames should use string-downcase, and patch the
  bad implementations.


Furthermore, notice what is said in section:
19.2.2.1.2.2 Common Case in Pathname Components

On a unix file system, we should have:

(make-pathname (pathname-host #P"/") :directory '(:absolute) :name "foo" :case :common)
--> #P"/FOO"

(make-pathname (pathname-host #P"/") :directory '(:absolute) :name "FOO" :case :common)
--> #P"/foo"

(make-pathname (pathname-host #P"/") :directory '(:absolute) :name "Foo" :case :common)
--> #P"/Foo"



(make-pathname "LOGICAL-HOST" :directory '(:absolute) :name "foo" :case :common)
--> #P"LOGICAL-HOST:FOO"

(make-pathname (pathname-host #P"/") :name "FOO" :case :common)
--> #P"LOGICAL-HOST:FOO"

(make-pathname (pathname-host #P"/") :name "Foo" :case :common)
--> #P"LOGICAL-HOST:FOO"

(setf (logical-pathname-translations "LOGICAL-HOST")
    (list (list "LOGICAL-HOST:*.*" "/tmp/*.*")
          (list "LOGICAL-HOST:*"   "/tmp/*")))
          
(translate-logical-pathname #P"LOGICAL-HOST:FOO")
--> #P"/tmp/foo" ; and definitely not #P"/tmp/FOO"

since:

    19.3.1.1.7 Lowercase Letters in a Logical Pathname Namestring

    When parsing words and wildcard-words, lowercase letters are
    translated to uppercase.

#P"LOGICAL-HOST:foo" : should return
--> #P"LOGICAL-HOST:FOO"

and not #P"LOGICAL-HOST:foo" as it does in ccl (and a few other
implementations).


People are complaining often about logical pathname, but the only
problem there is with them, is that they're almost never implemented as
specified…


One last note however: on unix you can mount foreign file systems, such
as MS-DOS FAT, MS-Windows, Mac HFS+, etc  (you can even mount file
systems in file systems, so a given path can contain components that
designates elements in multiple different file systems with different
conventions and "customary case"!

This means that converting:

#P"ROOT:ON-EXT2;HFS-MP;ON-HFS;DOS-MP;ON-DOS;EXT4-MP;ON-EXT4;FILE.TXT"
using the default mapping implied by customary case of each component
would give a physical pathname such as:
#P"/on-ext2/hfs-mp/On-Hfs/Dos-Mp/ON-DOS/EXT4-MP/on-ext4/file.txt" Also,
file systems can be mounted and unmounted dynamically, so another
translations at another time could give a different rule.  And we you
also have to translate pathnames to inexistant files, and therefore use
the customary case of the last file system seen for the inexistant
parts).  (There's no such #P"/tmp/FOO"  file on my file system, but
#P"/tmp/" exists and has the lower customary case, so
#P"LOGICAL-HOST:FOO" could still be translated to #P"/tmp/foo").



And finally, it should be noted that most implementation provide hooks
to let the user specify the mapping between module names and modules and
therefore even if you provide a sane mapping and implementation here,
nothing would prevent the user to implement a case insensitive mappting
or something stranger.  But he would control it, so up to him.


https://groups.google.com/forum/#!search/comp.lang.lisp$20pjb$20logical$20$20pathnames$20implementations/comp.lang.lisp/kA5KT4WNF8k/1CSM8Hla6xAJ

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

case sensitivity of provide, require and module-provide-asdf

[Daniel J. <dan...@gm...>]

While updating the asdf module to 3.1.7, I found that our provide and
require functions are actually case sensitive. Given the ignorance of case
prevalent in Common Lisp, I was surprised that the HyperSpec doesn't
contain any notes about case sensitivity for provide and require, thus I'd
assume that we're conforming to ANSI here, right?

All of this originated from the surprise when running CLISP with the new
asdf, because asdf now provides both "ASDF" and "asdf" (as well as "UIOP"
and "uiop"). Excerpt from asdf.lisp:

;; Provide both lowercase and uppercase, to satisfy more people,
especially LispWorks users.
(provide "asdf") (provide "ASDF")

Consequently, *modules* now also contains both the lowercase and the
uppercase names.

So far so good, where it really gets surprising is in the registered module
provider function asdf/operate::module-provide-asdf, because that function
(part of asdf.lisp) just downcases the module name right away.

Thus, (require "foo"), (require "foO") and (require "FOO") will each try to
load (via asdf) the module "foo". *modules* will only contain what the module
actually provides, though.

This feels very inconsistent to me. Some approaches:

* Leave as it is.
* Make provide and require case insensitive.
* Let asdf only provide either of "ASDF" or "asdf". Though this is more of a
cosmetic fix.
* Ask asdf about it.

What are your thoughts on this?

Re: GSoC status report

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-12 03:25:49 +0200]:
>
> CLISP builds with libsigsegv, too.

does GENERATIONAL_GC get enabled?

> The tests run exactly the same as without libsigsegv: 2 errors for
> tests/socket.tst and 3 for tests/path.tst.

this is weird, I don't remember such failures.

> Jörg wrote (regarding "64 bit support"):
>> I'm confused. If on amd64, you're building a 64bit version, aren't you?
>> So what doesn't get configured and what builds and passes tests?
>
> (on linux)
> I'm on amd64, and am building a 64bit version. Both clisp and lisp.run are
> 64bit elf executables. There's :WORD-SIZE=64 in *features*.

(integer-length most-positive-fixnum) should return 48.

> Instead of using defaults, I passed --without-ffcall, --without-dynamic-modules
> and --without-readline (mostly out of curiosity). On linux amd64, this fails
> with:
>
> (CHECK-OS-ERROR (SOCKET-SERVER 1240 :INTERFACE "[/]=") (:EINVAL 22))
> [OS-ERROR]:
> *** - handle_fault error2 ! address = 0x0 not in [0x33437f000,0x3346e2dc0) !
> SIGSEGV cannot be cured. Fault address = 0x0.
>
> This looks a lot as if the message it's about to print isn't a valid
> (C) string.

yes, looks like it's a NULL pointer error.

> Summarizing the configuration related stuff:
>
>  * libsigsegv, ffcall, dynamic-modules, unicode, readline: yes
>  * threads, jitc: no
>  * disabling mmap on NetBSD is ok
>  * garbage collector / object representation: leave as is

yes, but note platforms were you cannot build GENERATIONAL_GC.

>  * check that fixnums have 48 bits available on 64bit architectures (?)
>  * configuration system itself remains on TODO list for later

> Ok, I'll try to bring some news regarding this ASAP. asdf should be updated
> to it's recent version, right? (there was a post from its maintainer on
> clisp-list: https://sourceforge.net/p/clisp/mailman/message/35010071/ )

yes

> ./configure --help-modules
> module sets found in the directory 'modules':
> find: `modules': No such file or directory
> to specify the location of external software:

this is weird, never seen this.

BTW, I suggest that you build in separate directories.
(by passing dir arg to the top-level configure)

> Sam wrote:
>> You will have to update CLISP with 6 years worth of gnulib updates, and
>> also figure out how to make it work on windows.
>> This is the meat of the project.
>> This is what the midterm eval will hinge on.
>
> Is there any "list" (possibly from the time of the first gnulib import) of
> what gnulib module is imported for what reason?
> In order to get anything to work on windows I'll find and fix that pathname
> issues now.

Makefile.devel is your friend.
In a way, your only friend :-(
Search for the word release there.

> Finally: Do these mails get too long? Would it be better to split
> further discussion by topic, i.e. windows-pathname related,

My personal preference is for short emails with specific questions and
clear subject lines.

However, Jörg has an opposite preference, and, at any rate, it is really
up to you.

--
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://mideasttruth.com http://thereligionofpeace.com
http://dhimmi.org http://iris.org.il http://honestreporting.com
If a train station is a place where a train stops, what's a workstation?

Re: GSoC status report

[<Joe...@t-...>]

Daniel Jour wrote:
>I figured it would be good to know how reliable CLISP itself is before approaching the modules.
Very reasonable. If CLISP crashes in GC or sockets, there's no value in modules.
However, a quick & early shot at modules is not wrong either, to give you
an estimate of expected trouble. OTOH, I wouldn't throw a debugger at modules
if I'm not 100% certain that base CLISP is in perfect shape.

>if HAVE_LONG_LONG_INT is not defined [...].
>Thus I'd assume it's some "configure" bug, though I'm with Sam here:
Scan the output of configure and see if it looks reasonable or suspicious.
FFI users won't be happy without 64bit ints.

>Do these mails get too long? Would it be better to split further discussion by topic
More mails means more to click vs. just moving the scrollbar. YMMV
If you feel that some topic has the potential to need many mails, split it off.

>I mean the Lisp level tests (tests/ffi.tst). Currently I'm only using the
>binary package from the distribution
>(libffcall-2009-05-27 ... wow, that's kind of dated.)
Let's hope that the package maintainers ran the ffcall testsuite.
There have been patches to ffcall floating around in the last years, but perhaps
they all were about MacOSX support, which is not on your GSoC list.

Sam wrote:
>You need to build at least the following modules: [...] new-clx
Presumably stumpwm uses it? That's likely an important CLISP application.
https://wiki.archlinux.org/index.php/stumpwm shows that it needs new-clx, asdf and ppcre,
and that it also builds with sbcl.

Regards,
	Jörg

Re: GSoC status report

[Daniel J. <dan...@gm...>]

Jörg wrote:
> Did you mean the Lisp level FFI tests or the C tests part of the libffcall
> library?
> Did you build the ffcall library on your own or use the binary package
> from the distribution?

I mean the Lisp level tests (tests/ffi.tst). Currently I'm only using the
binary package from the distribution
(libffcall-2009-05-27 ... wow, that's kind of dated.)


Jörg wrote:
> +2 A CLISP release should not be built without libsigsegv. Libsigsegv is
> required for generational GC, and a CLISP without genGC is an order of
> magnitude slower than when it's available.
> OTOH, better without genGC than no CLISP at all. YMMV.
>
> IIRC, early binaries for MS-Windows ran without libsigsegv.
> Actually, has libsigsegv ever been ported to MS-Windows?

I mostly built without libsigsegv because of lazyness to download it.
Windows (cygwin, mingw, as it seems also "pure native" msvc) seems to be
supported, according to the PORTING file. And indeed, it builds and passes
its tests (it's reporting i686-pc-mingw32 as detected target, which is correct
but surprised me due to running on win 10 amd64 with MinGW64).

CLISP builds with libsigsegv, too. Though interestingly it adds even more
issues with regards to pathnames (without libsigsegv only the argument of
a Lisp level load call had to be adjusted, with libsigsegv also the -lp
parameter to lisp.exe)
The tests run exactly the same as without libsigsegv: 2 errors for
tests/socket.tst and 3 for tests/path.tst.

Regarding the socket tests: At first I thought it could be the firewall, but
even without it trying to communicate with a socket server just hangs.

Adding this link for pure reference, someone who built on mingw, too:
http://www.frank-buss.de/lisp/clisp.html

Jörg wrote (regarding "64 bit support"):
> I'm confused. If on amd64, you're building a 64bit version, aren't you?
> So what doesn't get configured and what builds and passes tests?

(on linux)
I'm on amd64, and am building a 64bit version. Both clisp and lisp.run are
64bit elf executables. There's :WORD-SIZE=64 in *features*.
Running

(use-package "FFI")
(WITH-C-VAR (X 'SINT64 1229782938247303441) T)

results in

[SIMPLE-ERROR]: FFI::EXEC-ON-STACK: 64 bit integers are not supported
on this platform and with this C compiler: SINT64

>From a first glance at the relevant code, this message can only show up if
HAVE_LONG_LONG_INT is not defined (in src/foreign.d, because then error_64bit
is called from convert_to_foreign). Thus I'd assume it's some "configure" bug,
though I'm with Sam here:

Sam wrote:
> not too good, but I sure we can figure this out eventually.



Jörg wrote:
> "Downstream" means the NetBSD CLISP package maintainer?
> Mmap issues can get very hairy and very low-level fast. CLISP tries to put
> its typecodes within the 32 bit pointers and may soon get in conflict with the
> OS memory layout. I don't know how ASLR works with CLISP. I'd expect much less
> mmap trouble with a 64bit build.

I found this: http://gnats.netbsd.org/33189
They just add a flag --disable-mmap. From a few first tests it seems that
MAP_FIXED (which the offending function, mmap_zeromap, uses) is entirely
unuseable on NetBSD. I'll try to get more information on this from their
mailing list.

Btw, is there any technical reason for using a fixed address instead of
letting the underlying OS choose a suitable location?

Jörg wrote:
> (Note that I never put my hands on a NetBSD system.)

When I read "Pullup ticket 1670 - requested by joerg" (in above link) I hoped
that "joerg" would be you, but I guess I'm wrong then :)


Sam wrote:
>>  * linux, ./configure (tried a minimum config): builds, tests segfault
>>  when trying to run a socket server, possibly due to a corrupted error
>>  message.
>
> I am confused: how is this different from above?

Instead of using defaults, I passed --without-ffcall, --without-dynamic-modules
and --without-readline (mostly out of curiosity). On linux amd64, this fails
with:

(CHECK-OS-ERROR (SOCKET-SERVER 1240 :INTERFACE "[/]=") (:EINVAL 22))
[OS-ERROR]:
*** - handle_fault error2 ! address = 0x0 not in [0x33437f000,0x3346e2dc0) !
SIGSEGV cannot be cured. Fault address = 0x0.

This looks a lot as if the message it's about to print isn't a valid
(C) string.


Sam wrote:
> You did not even scratch the surface, sorry.

I figured it would be good to know how reliable CLISP itself is before
approaching the modules.
Summarizing the configuration related stuff:

 * libsigsegv, ffcall, dynamic-modules, unicode, readline: yes
 * threads, jitc: no
 * disabling mmap on NetBSD is ok
 * garbage collector / object representation: leave as is
 * check that fixnums have 48 bits available on 64bit architectures (?)
 * configuration system itself remains on TODO list for later


Sam wrote:
> The real problem is, as I said before, with gnulib files in modules.
>
> You need to build at least the following modules:
> base (i18n, regexp, syscalls, readline)
> full (rawsock, bindings, asdf, pcre, new-clx)
> all others for which you can _easily_ get the back end (berkeley-db,
> fastcgi, gdbm, libsvm, pari, postgresql, zlib)

Ok, I'll try to bring some news regarding this ASAP. asdf should be updated
to it's recent version, right? (there was a post from its maintainer on
clisp-list: https://sourceforge.net/p/clisp/mailman/message/35010071/ )


Sam wrote:
> Many modules come with their own test suites which can be run with "make
> mod-check", and I bet rawsock will fail on windows spectacularly.

I just tried with rawsock and indeed, it didn't come very far, though that's
not really related to rawsock: Running make mod-check recompiles
(and breaks!) clisp.exe leading to another pathname related issue.

Moreover, it didn't even build rawsock because (again, pathnames) it didn't
find the modules folder:

./configure --help-modules
module sets found in the directory 'modules':
find: `modules': No such file or directory
to specify the location of external software:


Sam wrote:
> You will have to update CLISP with 6 years worth of gnulib updates, and
> also figure out how to make it work on windows.
> This is the meat of the project.
> This is what the midterm eval will hinge on.

Is there any "list" (possibly from the time of the first gnulib import) of
what gnulib module is imported for what reason?
In order to get anything to work on windows I'll find and fix that pathname
issues now.


Jörg wrote:
> Not needed with SLIME or withint Emacs, but I don't know whether current SLIME
> still supports CLISP (I haven't heard about CLISP in the SLIME mailing list
> for a long time).

Pascal wrote:
> Clisp works ok with slime.

I am using this setup since about a year without major issues.


Finally: Do these mails get too long? Would it be better to split further
discussion by topic, i.e. windows-pathname related, HAVE_LONG_LONG_INT
related, ... ?

Re: GSoC status report

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-10 02:27:03 +0200]:
>
> Sam wrote:
>> There are so many such bugs in CLISP that Daniel will be able to spend
>> the whole summer on them.
>> Fixing them requires an effort: investigating when they were introduced,
>> testing the fix, &c &c.
>> This is important work, but it is out of the project's scope.
>
> The "main goal" is getting a release done. For that CLISP should build and
> pass its self tests on the "supported platforms" (linux, bsd, windows - more
> is better), right?
>
> Do we have any bugs in the bug tracker that must be considered
> "release critical",
> that is there cannot be a release without them being fixed? (I'm trying to
> build a list of things "to do".)

Probably none.

> Sam wrote:
>> Daniel has a huge task ahead of him:
>>
>> He has 6 weeks until June 20, 2016 (Mid-Term Evaluation) to pass
>> the first hurdle: build CLISP on
>> 1. linux (one flavor),
>> 2. bsd (one flavor),
>> 3. windows (mingw?).
>
> I think we should also define "build" a bit better, since CLISP can be build
> in quite a few different configurations. Of the top of my head there's
>
> * ffcall

yes, this is mission-critical for many users.

> * dynamic modules

yes, this is mission-critical for many users.

> * unicode

yes, this is extremely useful.

> * readline

yes, this is not hard to get working and it is useful.

> * threads (POSIX, WIN32, SOLARIS?)

no, this is still "experimental" (IOW, causes segfaults under known
conditions).

> * jitc (lightning)

no, I don't think this ever became "usable"

> Then, CLISP can also be build without libsigsegv.

libsigsegv is required for Generational GC.
it's an absolute must have.
the reason --ignore-absence-of-libsigsegv is even there is for initial
porting work. any serious use of clisp requires libsigsegv.

> Btw: Are there any known "out of tree" modules for CLISP?

Fred and Don Cohens might have some, but let's not worry about these for
now.

> Regarding configurations: What about the different garbage collectors / object
> representations? (TYPECODES vs HEAPCODES)

I would leave this to configure, except for ...

> What about "64bit support"?

... Fixnums must be 48 bits on any 64-bit architecture.

> Next, a short status report:
> Apart from reading further through the code I've now setup everything I need
> to build on linux, netbsd (freebsd comming, too) and windows (10 with mingw).
> All of these are running on amd64, I plan to add x86 VMs soon.

sounds good.

> I also tried to build CLISP on these platforms, naturally. The results:
>
>  * netbsd, ./configure, (FFI: no, readline: no, sigsegv: yes): build fails
>  due to an issue with mmap, though this seems to be "fixed" downstream by
>  disabling mmap support.

disabling mmap is fine.
FFI is not so much, but this is a lower priority platform.

BTW, please take a look at clisp/unix/PLATFORMS and clisp/unix/INSTALL.

>  * linux, ./configure, (defaults) with gc checker: builds, passes
>  tests except ffi 64 bit (HAVE_LONG_LONG seems to be undefined for

not too good, but I sure we can figure this out eventually.

>  whatever reason).  Same without gc checker.

Cool.

>  * linux, ./configure (tried a minimum config): builds, tests segfault
>  when trying to run a socket server, possibly due to a corrupted error
>  message.

I am confused: how is this different from above?

>  * windows, ./configure (FFI: no, readline: no, sigsegv: NO): builds

sigsegv - must have.

>  with some minor manual intervention (path handling issues), passes
>  tests except path handling (though this might be related to MinGW,
>  not sure if this is really a bug) and a socket failure (though this
>  could be caused by the firewall going off during the test).

fine.

> In general, the configure / build "system" seems to be a bit (emergency)
> patched. Some parts are "handwritten" ... why not use (all of) autotools?
> (this would also allow easy integration with libtool, btw)

this has been on the TODO list for a long time ...

> Sam wrote:
>> If he cannot build CLISP on Windows in 6 week, this is what will happen.
>
> I know that this is currently rather vague, but too me it doesn't look
> too bad atm, most of CLISP seems to work (though thinking of some of
> the scarier parts of the code this really surprises me :D This code
> really "evolved/grew" over time ... it's funny that some of it is
> older than me).

You did not even scratch the surface, sorry.

The real problem is, as I said before, with gnulib files in modules.

You need to build at least the following modules:
base (i18n, regexp, syscalls, readline)
full (rawsock, bindings, asdf, pcre, new-clx)
all others for which you can _easily_ get the back end (berkeley-db,
fastcgi, gdbm, libsvm, pari, postgresql, zlib)

Many modules come with their own test suites which can be run with "make
mod-check", and I bet rawsock will fail on windows spectacularly.

You will have to update CLISP with 6 years worth of gnulib updates, and
also figure out how to make it work on windows.
This is the meat of the project.
This is what the midterm eval will hinge on.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://jihadwatch.org http://iris.org.il
http://thereligionofpeace.com http://camera.org http://americancensorship.org
Conscience is like a hamster: it is either asleep or gnawing.

Re: GSoC status report

[Pascal B. <pj...@in...>]

I would add:

There are definitively a few clisp modules that have been developed by users; I've seen mention of one or two in cll over the years IIRC. 

Clisp works ok with slime. If there have not been recent mentions of it, it's both because for the lack of mainenance less users have been using it with slime and because the current support is adequate. 

I don't think jitc is production ready. 

-- 
__Pascal Bourguignon__

Re: GSoC status report

[<Joe...@t-...>]

Hi,

Daniel Jour wrote:

>AFAIK the FFI is a pretty important piece of CLISP, so I guess it should "stay", right?
Yes. It seems to me like most people have a use for it.

>(It's - according to the self tests - working, at least on linux)
Did you mean the Lisp level FFI tests or the C tests part of the libffcall library?
Did you build the ffcall library on your own or use the binary package from the distribution?

>Are there any known "out of tree" modules for CLISP?
You mean written by other people and not in the CLISP source? I can't remember of
any such mention in the mailing list. Presumably, people just used the ability
to dynamically load a foreign library (as in other Lisp implementations),
without creating a module structure.


>CLISP can be build in quite a few different configurations.
>* ffcall
+2 See above

>* dynamic modules
There used to be a possibility to use that without ffcall, but nowadays the
Lisp packages seem to use a mixture of both. The low-level stuff for dynamic
modules (linking of object files) should not be problematic, but individual
modules may have trouble interfacing with their respective libraries as the
latter ones will have evolved (e.g. libSVN, Oracle, GTK etc.) during the
last 6 years.

Effort here very much depends on how many other packages you are ready to
install, then test.

These should be unproblematic (but please ask Sam, he knows better), thanks of
few or old dependencies:
 +1 rawsock (mentioned every so often in the mailing list)
 +1 FastCGI
 + POSIX RE
 + zlib (highly subjective)

 ~ syscall may be important to several users, but may need updates, as the Linux kernel evolved;
 ~ PCRE certainly got a new API meanwhile;
 ? Oracle
 ? Postgres
 ...

>* unicode
+2 I believe this needs libiconv (except on the MS platform). Probably it doesn't
Make sense in 2016 to release something that does not support encodings.
I can't remember much trouble building with encodings.

>* readline
+1 Excellent for comfort. Has never been a problem getting it to run AFAICR.
Not needed with SLIME or withint Emacs, but I don't know whether current SLIME
still supports CLISP (I haven't heard about CLISP in the SLIME mailing list
for a long time).

>* threads (POSIX, WIN32, SOLARIS?)
Still not 100% ready, issue with hashtables IIRC. Don't try and debug this for now,
that would be another project IMHO.

>* jitc (lightning)
Can't remember, was that subproject led to completion?


>Then, CLISP can also be build without libsigsegv. What configurations are the most interesting ones?
+2 A CLISP release should not be built without libsigsegv. Libsigsegv is required
for generational GC, and a CLISP without genGC is an order of magnitude slower
than when it's available.
OTOH, better without genGC than no CLISP at all. YMMV.

IIRC, early binaries for MS-Windows ran without libsigsegv.
Actually, has libsigsegv ever been ported to MS-Windows?


>Regarding configurations: What about the different garbage collectors /
>object representations? (TYPECODES vs HEAPCODES)
Just let configure do its magic. IIRC, one is less troublesome but less
performant in the 32bit world, and the other next to mandatory in a 64bit
world (there's a lot of room in 64 bits to store typecodes).


>a socket failure
There were/are a few bugs related to sockets reported to the mailing list.
(sometimes due to compiler errors?)


>What about "64bit support"? (I'd consider that important, but it's currently "broken"
>in the sense that it doesn't get configured even when the target would support it.)

>All of these are running on amd64, I plan to add x86 VMs soon.

>* linux, ./configure, (defaults) with gc checker: builds, passes tests  except
>ffi 64 bit (HAVE_LONG_LONG seems to be undefined for whatever reason).
> Same without gc checker.

I'm confused. If on amd64, you're building a 64bit version, aren't you? So what
doesn't get configured and what builds and passes tests?


>* netbsd, ./configure, (FFI: no, readline: no, sigsegv: yes): build fails  due to an
>issue with mmap, though this seems to be "fixed" downstream by  disabling mmap support.
"Downstream" means the NetBSD CLISP package maintainer?
Mmap issues can get very hairy and very low-level fast. CLISP tries to put
its typecodes within the 32 bit pointers and may soon get in conflict with the
OS memory layout. I don't know how ASLR works with CLISP. I'd expect much less
mmap trouble with a 64bit build. (Note that I never put my hands on a NetBSD system.)

Oh, please read what I write with a grain of salt. It's all from memory;
I haven't compiled CLISP in the last ~10 years.

Regards,
	Jörg Höhle

GSoC status report

[Daniel J. <dan...@gm...>]

A good portion of "community bonding time" is already over, so I guess it's
a good time for a new status report, along with some new questions :)

I start with the "heavy" questions:

Jörg wrote:
> I'm not saying that the bugs in the bug tracker don't need prioritization
> as a general matter.

Sam wrote:
> There are so many such bugs in CLISP that Daniel will be able to spend
>> the whole summer on them.
> Fixing them requires an effort: investigating when they were introduced,
> testing the fix, &c &c.
> This is important work, but it is out of the project's scope.

The "main goal" is getting a release done. For that CLISP should build and
pass its self tests on the "supported platforms" (linux, bsd, windows - more
is better), right?

Do we have any bugs in the bug tracker that must be considered
"release critical",
that is there cannot be a release without them being fixed? (I'm trying to
build a list of things "to do".)

Sam wrote:
> Daniel has a huge task ahead of him:
>
> He has 6 weeks until June 20, 2016 (Mid-Term Evaluation) to pass
> the first hurdle: build CLISP on
> 1. linux (one flavor),
> 2. bsd (one flavor),
> 3. windows (mingw?).

I think we should also define "build" a bit better, since CLISP can be build
in quite a few different configurations. Of the top of my head there's

* ffcall
* dynamic modules
* unicode
* readline
* threads (POSIX, WIN32, SOLARIS?)
* jitc (lightning)

Then, CLISP can also be build without libsigsegv. What configurations are the
most interesting ones?

Jörg Höhle wrote:
> This would p.o. lots of people, but you could consider building clisp
> without FFI, that's quite a task already. Please discuss priorities with
> your mentor.

AFAIK the FFI is a pretty important piece of CLISP, so I guess it should
"stay", right? (It's - according to the self tests - working, at least on linux)
Btw: Are there any known "out of tree" modules for CLISP?

Regarding configurations: What about the different garbage collectors / object
representations? (TYPECODES vs HEAPCODES)

What about "64bit support"? (I'd consider that important, but it's currently
"broken" in the sense that it doesn't get configured even when the target
would support it.)


Next, a short status report:
Apart from reading further through the code I've now setup everything I need
to build on linux, netbsd (freebsd comming, too) and windows (10 with mingw).
All of these are running on amd64, I plan to add x86 VMs soon.

I also tried to build CLISP on these platforms, naturally. The results:

 * netbsd, ./configure, (FFI: no, readline: no, sigsegv: yes): build fails
 due to an issue with mmap, though this seems to be "fixed" downstream by
 disabling mmap support.
 * linux, ./configure, (defaults) with gc checker: builds, passes tests
 except ffi 64 bit (HAVE_LONG_LONG seems to be undefined for whatever reason).
 Same without gc checker.
 * linux, ./configure (tried a minimum config): builds, tests segfault when
 trying to run a socket server, possibly due to a corrupted error message.
 * windows, ./configure (FFI: no, readline: no, sigsegv: NO): builds with some
 minor manual intervention (path handling issues), passes tests except path
 handling (though this might be related to MinGW, not sure if this is really
 a bug) and a socket failure (though this could be caused by the firewall going
 off during the test).

In general, the configure / build "system" seems to be a bit (emergency)
patched. Some parts are "handwritten" ... why not use (all of) autotools?
(this would also allow easy integration with libtool, btw)

Sam wrote:
> If he cannot build CLISP on Windows in 6 week, this is what will happen.

I know that this is currently rather vague, but too me it doesn't look too
bad atm, most of CLISP seems to work (though thinking of some of the scarier
parts of the code this really surprises me :D This code really "evolved/grew"
over time ... it's funny that some of it is older than me).


Sam wrote:
> I don't want you to lose focus.

That's something I'm also concerned of, so I'm paying extra attention to not
digress. If you notice me losing time on something that's not critical for
the release, then just tell me :)

Re: multiple values: value1 treated extra even when not in register

[Daniel J. <dan...@gm...>]

Sam wrote:
> There will be so many such minor things that you will be able to spend
> the whole summer on them.

Jörg wrote:
> I disagree with the suggested treatment.  Daniel found a bug, all by
> himself.  The best reward is to let him proudly fix it immediately, so
> he can forget about this particular task and lessen the burden on his
> brain.  I believe the direct reward to be more motivating than
> "maybe in a couple of months".

Thank you, though it's not really about the "reward" but more about
understanding "why not": It's a bug that, by itself, causes no harm. It
could have lead to serious headaches when hunting a possible other bug in
the multiple value code, because one does not expect value1 to be treated
any different when it's not in a register. Thus, it could have happended
that I'd looked at the wrong code paths (the one without extra treatment).
Regarding the time: It's already spent, and it didn't cost much "extra"
because what I was actually doing when I found it was reading the code in
order to understand how all (or better: the core parts) of it works.

Sam wrote:
> He should create a bug report or a patch and leave it in the SF bug
> tracker for later.

That's what I'll do for now, then. (The change is already applied in the
forked hg repo, though. I guess reverting it to later bring it back in might
be a bit odd.)

https://sourceforge.net/p/clisp/patches/45/

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Daniel Jour wrote (in March):
>* Undefined behavior, probably mostly in spvw* and also sort.d:
>Stuff like comparing (less, greater) pointers, out of bounds indexing and
>some cast just aren't legal C, and can lead to all sorts of nasty bugs,
>especially when using an optimizing compiler. I'd guess that's the reason
>that (e.g.) my distributions build instructions pass in CFLAGS=-O0.

-O0?!? That feels like a slap in the face. Decades ago, we spent countless
hours looking at assembly output and optimizing this and that, measuring
time. We felt that even small performance improvements (i.e. ~2%) were
worth it. That's the origin of the dotimespL() macros. We were very proud
when something produced a 5% improvement. Bruno regularly scrutinized the
bytecode interpreter loop, which is the #1 hot spot.

IIRC, parts of the old libffcall would not even work with -O0, because it
relied on some stack optimizations by GCC, like -fomit-frame-pointer.
(Nowadays, I don't know whether libffcall still makes use of the asm
 files, or compiles even the lowest-level stuff from C).

-O0 output is so pitiful to look at.

Regarding the recent changes in C compilers, I believe it would be
preferable to use, like several other projects, e.g. the Linux kernel or
PostgreSQL, options like -fno-strict-overflow (or -fwrapv),
-fno-delete-null-pointer-checks, -fno-strict-aliasing and whatever else
in order to use optimizations yet behave like all C compilers used
to behave during the last decades. Later, one can add additional casts,
wrapped in yet more macros...;-) to compare pointers of different kinds.

Sam Steingold wrote:
> I see no problem in relying on external libraries.
Nowadays. Initially, Bruno reported that lots of libraries on the many UNIX
systems that CLISP ran on had lots of different bugs, and he was not amused.
As a result, when I did the Amiga port, he advised that we avoid any
lib, thus Amiga-CLISP used nothing but (the equivalent of) syscalls, no
wrappers, and I even wrote the C startup code.
On UNIX, CLISP started using configure.

Regards,
	Jörg Höhle

Re: multiple values: value1 treated extra even when not in register

[Sam S. <sd...@gn...>]

Jörg,

> *  <Wbret-Plevy.Ubruyr@g-flfgrzf.pbz> [2016-05-06 11:52:09 +0200]:
>
>>> Why not include it in the release?
>>I don't want you to lose focus.
>>There will be so many such minor things that you will be able to spend the whole summer on them.
>
> I disagree with the suggested treatment.  Daniel found a bug, all by
> himself.  The best reward is to let him proudly fix it immediately, so
> he can forget about this particular task and lessen the burden on his
> brain.

He should create a bug report or a patch and leave it in the SF bug tracker for later.

> I believe the direct reward to be more motivating than
> "maybe in a couple of months".

This "direct reward" may cost him a failed project.
If he cannot build CLISP on Windows in 6 week, this is what will happen.

> I'm not saying that the bugs in the bug tracker don't need prioritization as a general matter.

There are so many such bugs in CLISP that Daniel will be able to spend
the whole summer on them.
Fixing them requires an effort: investigating when they were introduced,
testing the fix, &c &c.
This is important work, but it is out of the project's scope.

Daniel has a huge task ahead of him:

He has 6 weeks until June 20, 2016 (Mid-Term Evaluation) to pass
the first hurdle: build CLISP on
1. linux (one flavor),
2. bsd (one flavor),
3. windows (mingw?).

Any second spent on anything else increases the probability of a failure.

Minor bugs can be fixed later in the Summer of even in the Fall for the
3.01 release.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://truepeace.org http://iris.org.il
http://americancensorship.org http://thereligionofpeace.com http://dhimmi.org
If you cannot improve on silence, keep it.

Re: clisp - general interest in code cleanup

[Pascal J. B. <pj...@in...>]

<Joe...@t-...> writes:

> it in a positive way) view on what's actually important. As a result,
> those days, programmers view code in a tinier window than when I
> started computing on a 9" screen over 30 years ago.
> That's no progress to me!

LOL. Indeed.

>> Something like the Smalltalk
>> code browser: there's no notion of file then.
>
> It works nicely for Smalltalk, because unlike C or Common Lisp, the
> semantics of a method body does not depend on a myriad of definitions
> that appears in lexical order before it in the source file.
>  - No macros nor any variations to the syntax;
>  - Nothing like operator precedence (except unary operators);
>  - class and instance variables are one window away.
> Hence IMHO, any parallel is not applicable.

Reader macros can be treated as a dependency (at the form level).

But you're right that a requirement for an IDE nowadays is to include an
execution environment where the source being edited is also compiled and
loaded.  In modern Java IDE and C/C++ (with clang) IDE, they compile the
source at least.  In lisp we'd have to add the run-time environment, to
be able to identify correctly more elements, notably if we don't have a
global static analyser.  This is what slime does.

Granted it's possible to write lisp programs that will fight even this
step, but remember that macros and reader macros should be idempotent,
so there's already some provision in the standard to help IDE there.
Also, with the promotion of ASDF, package and system set up has become
more declarative (rather than procedural); modern lisp code is more
compatible with an IDE treatment.


> In summary, the 'function at a time' might work with programs with
> strong locality, e.g. tree structured. But not when you need to
> implement a collection of event handlers. Those hide the flow and
> ordering of messages and calls -- in Smalltalk too.

Indeed, I had such an example in a recent Android program I had to
maintain; it sent events to communicate between modules (assumedly on
different threads).  This kind of dependency could also be taken into
account by the IDE and it could provide navigation helpers, either
because (as in the case of this Android program) it uses a "standard"
communication mechanis, or because the programmer could declare such
dependencies to the IDE. Notably in lisp, we have the advantage of using
symbols that can be easily identified across modules.  Navigating to
uses of a given symbol could be all that is needed for the basic
feature.


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: GSoC 2016: proposal accepted!

[<Joe...@t-...>]

Daniel Jour wrote:
>Do you know whether there are "workarounds" around libffcall bugs in CLISP's code?
There shouldn't be. At the start, CLISP and ffcall were in the same
hands: Bruno Haible. One day, Bruno realized that other projects could
make use (or already had?) of the ffcall stuff and separated it as a
library that he maintained for several years. At some time,
the ffcall source moved out of the clisp source tree.

When the packages are maintained, all is well.  When not, your situation
is not unlike Debian, Suse, Ubuntu etc. maintainers' job: have some
patches (to libffcall, not clisp) at hand to keep them running...
It's best if the patches are on the bug tracker of the ffcall project.

This would p.o. lots of people, but you could consider building clisp
without FFI, that's quite a task already. Please discuss priorities with
your mentor.

Regards,
	Jörg Höhle

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Pascal J. Bourguignon wrote:

> [...] for a later project, we could imagine an emacs mode where
> each function (or toplevel form) is presented in its own buffer,
> with navigation buttons to the callers and callees and other
> references to the function.  

This remembers me of the Visual Basic Editor in MS-Word. It always
felt like an aberration to me (and I've heard others express similar
feelings), because of total lack of orientation, never knowing
whether there's perhaps another method, callback or event handler
that's needed to understand the code, hidden somewhere else, only
MS gurus know where!

Compare with a bottom up file: you have all in code in front of you.
Print it on paper for even better reading (I haven't done so for
long, dreaming of a paperless office, but you'll hear some well-known
SW-gurus really recommend this).

Perhaps I'm old-fashioned. The TV & monitors I've bought have grown
bigger each and every time. Other people seem to favour the opposite:
they work with Eclipse or a similar IDE, and with every release, it
feels like the code window gets smaller and smaller, because the
information windows to the left, to the right, to the top and to the
bottom grow, yielding a very narrow (let's say "focused" to express
it in a positive way) view on what's actually important. As a result,
those days, programmers view code in a tinier window than when I
started computing on a 9" screen over 30 years ago.
That's no progress to me!


> Something like the Smalltalk
> code browser: there's no notion of file then.

It works nicely for Smalltalk, because unlike C or Common Lisp, the
semantics of a method body does not depend on a myriad of definitions
that appears in lexical order before it in the source file.
 - No macros nor any variations to the syntax;
 - Nothing like operator precedence (except unary operators);
 - class and instance variables are one window away.
Hence IMHO, any parallel is not applicable.

In summary, the 'function at a time' might work with programs with
strong locality, e.g. tree structured. But not when you need to
implement a collection of event handlers. Those hide the flow and
ordering of messages and calls -- in Smalltalk too.

Regards,
	Jörg Höhle

Re: multiple values: value1 treated extra even when not in register

[<Joe...@t-...>]

Sam,

>> Why not include it in the release?
>I don't want you to lose focus.
>There will be so many such minor things that you will be able to spend the whole summer on them.

I disagree with the suggested treatment.  Daniel found a bug, all by
himself.  The best reward is to let him proudly fix it immediately, so
he can forget about this particular task and lessen the burden on his
brain.  I believe the direct reward to be more motivating than
"maybe in a couple of months".

I'm not saying that the bugs in the bug tracker don't need prioritization as a general matter.

Regards,
	Jörg Höhle

Re: multiple values: value1 treated extra even when not in register

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-05 00:43:53 +0200]:
>
>> I suggest that you make a note about this (e.g., by filing a bug or a
>> patch) and revisit it after you make the release.
>
> Why not include it in the release?

I don't want you to lose focus.

> It seems like a minor change with manageable consequences to me.

There will be so many such minor things that you will be able to spend
the whole summer on them.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://think-israel.org http://palestinefacts.org
http://openvotingconsortium.org http://islamexposedonline.com http://ffii.org
cogito cogito ergo cogito sum

Re: clisp - general interest in code cleanup

[Pascal J. B. <pj...@in...>]

Daniel Jour <dan...@gm...> writes:

> The issue is if one does not have the time to start reading from the top,
> but must/want to make a change now. If you jump right into spvw.d and
> start reading code, then there's a lot that you have to look up (and
> eventually "backtrack", if you made a false assumption).
>
> That's why I like "layered" approaches: You put low level code into a
> separate file and create an include file that contains only what you're
> supposed to be using in "higher" layers together with a extensive summary
> of how these functions and structures behave and are to be used.

I would not want to detract you from your clisp maintainance, but for a
later project, we could imagine an emacs mode where each function (or
toplevel form) is presented in its own buffer, with navigation buttons
to the callers and callees and other references to the function.  emacs
would then manage itself the storing of the toplevel forms in one or
more files files, you wouldn't have to care about it.  Something like
the Smalltalk code browser: there's no notion of file then.

> (Though that's not an argument for separate files ... all of this could
> of course live in the same file; I think it's the "summary" that's the
> critical thing which is missing for some parts of the CLISP code)

Don't complain, at least you have the specifications of this program!
:-)


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: GSoC 2016: proposal accepted!

[Daniel J. <dan...@gm...>]

Sam Steingold wrote:
> Emacs has a good ChangeLog mode.
> basically, if you use clisp/emacs/d-mode.el and type M-x
> add-change-log-entry RET, Emacs should DTRT.

Ah, that's helpful :)


Sam Steingold wrote:
> Sure, you can talk to them :-)
> I think they are doing pretty much what you are doing - fixing up for a
> release, merging in various patches &c &c

I'll do that. There seems to be no publicly visible conversation yet, so
I'll just try the mailing list. Do you know whether there are "workarounds"
around libffcall bugs in CLISP's code?


Regarding the C++ GC checker:
I've found the part in the impnotes (35.5.3. Run-time GC-safety checks)
but have not yet looked at the corresponding code. Though it builds and
runs (albeit very slow) with the checker enabled!

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

>> Startup code, C string manipulation and argument parsing live together
>>with memory management in src/spvw.d. This is especially concerning if the
>>"topics" are intermingled.
> It's always interesting when new, i.e. unbiased eyes look at old code. To
> me, having arg parsing and mm in the file called spvw = Speicherverwaltung
> = memory management was logical. This (set of) files contain(s) all the
> low-level stuff that help create the Lisp world (of objects) that the
> other files all use.

When I read Speicherverwaltung (I'm unsure whether you know: I'm German :) ),
then what comes first to my mind is "malloc" and "free".
Thus, I'm basically expecting some sort of interface where I can request (and
free, normally) pieces of memory that satisfy some requirements (like size,
alignment) and which I can then use to create "objects" in them.
This view is probably heavily influenced by C++, though.

I also always favoured separation of "startup" code and "normal operation"
code. (When working on a OS kernel, that separation helps to get rid of the
former once the system is up)


>>Why didn't you run the concatenation as a build step?
> IIRC, Bruno's Makefile did this. I didn't, because of disk space. A 1MB
> file on a 80MB drive was a lot, so I preferred to work with temporary
> files, i.e. only keep .d and .o on disk.
>
> Actually, I can't remember. Perhaps for exactly the reasons mentioned
> previously: comfort of working with basically one include (lispbibl.d)
> and one code (e.g. spvw.d) file, each in one window of my (Emacs),
> Bruno's (axe) editor or whatever people used. Emacs' incremental search
> is often more effective than any combination of grep I know, even if
> you add some context lines with -A or -B (sometimes grep is better,
> e.g. to show all matches in tabular form).

I always forget how limited memory once was :) While reading further through
CLISP's code I found Emacs' search to be more effective than grepping at times
when I wanted to go to the previous or next occurence, starting from the
current point.


> Compare this with Linux includes, which are often annoyingly deep. I simply
> view this as different design choices. Bruno preferred to e.g. cover all
> variations of UNIXisms next to each other in unix.d and spvw.d, others
> prefer lots of different files. Each choice makes some operations easy
> and others difficult.

Searching through Linux includes can be a real pain, indeed. I'm glad to
be able to experience what it's like working with the other "extreme".


>>Though I don't understand what you mean with the forward references?
> You should first define types, then use them in function declarations;
> first define low-level functions, then the callers of these. That's
> the typical Pascal pyramidal ordering of program elements. For some
> programs and data structures you can't preserve this hierarchical
> ordering, you need forward references (IIRC, the Pascal programming
> language has them too).

So (regarding C) you mean forward declarations of structs and functions,
and you opt against them when possible (by keeping code that references
functions/structures "beneath" them)? That's a good approach IMO, and it
also works well with small files (albeit it's sometimes difficult to break
cyclic dependencies).
The way you describe that reminds me of "bottom up" programming (as
described by Paul Graham).


> CLISP code is ordered like much this and I fully agree with Bruno who
> once told me that it makes reviewing code easier as well as more robust.
> I believe the reason is that it avoids backtracking, and the human brain
> is particularly bad at backtracking, i.e. forgetting false assumptions
> about behavior of code.

Yes, forgetting a false assumption about a piece of code can be hard. OTOH
I found it sometimes to be helpful to not know exactly what some function
does but to first see how it's used: This made it easier to understand the
function itself.


> When you look at a function and see some call, it helps enormously to
> *know* what the function does *exactly* -- because you already reviewed
> it, reading from top to bottom through the file -- than to have to guess
> from the name or the 10 line header description whether it may cover
> this or that corner case, e.g.
>  - is the second parameter allowed to be NULL,
>  - is the upper bound inclusive or not,
>  - what if the array is empty,
>  - does it assume that lock X is held,
>  - will it release lock X in case of error, etc.

Reading from top to bottom through the file ... that reminds me a bit of
literate programming (I tried this only in Haskell). I guess it's a bit
like reading a book: You don't want to have to jump 100 pages ahead in
order to understand the current chapter. Structuring code like that is
IMO good, though it only works if one really starts reading from the top.

The issue is if one does not have the time to start reading from the top,
but must/want to make a change now. If you jump right into spvw.d and
start reading code, then there's a lot that you have to look up (and
eventually "backtrack", if you made a false assumption).

That's why I like "layered" approaches: You put low level code into a
separate file and create an include file that contains only what you're
supposed to be using in "higher" layers together with a extensive summary
of how these functions and structures behave and are to be used.
(Though that's not an argument for separate files ... all of this could
of course live in the same file; I think it's the "summary" that's the
critical thing which is missing for some parts of the CLISP code)