clisp-devel

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

Hi Daniel,

> * Daniel Jour <qna...@tz...> [2016-03-21 15:07:02 +0000]:
>
> the CLISP project seems to be "not that active", judging from the mailing
> list / bug tracker / repository activities. The latest release is 6 years
> old.

Alas, this is, indeed, the case.

> Looking at the code base, I think this low activity stems at least in part
> from the IMO relatively bad condition it is in. Most of it seems to be
> quite old, there are (now talking of the C parts) parts that invoke
> undefined behavior (general pointer comparisons, integer overflows, out of
> bounds indexing) and some other odd things (the whole additional
> preprocessing, pre C99 code, indexing from 1 instead of 0, apparently no
> internal interfaces / abstraction, probably more).
>
> I'd like to change that. In order to do so I'd like to "modernize" the code
> base, starting with the C code. That means among other things converting to
> standard compliant code (C99 probably), splitting code into smaller units,
> establishing internal interfaces.
>
> Before putting (a lot) more work into this I'd like to know whether
> a) there's is a general interest to make such - probably huge - changes to
> the code base, and whether

Yes, there is at least some interest in modernizing CLISP code base.
Note, however, that this is a fairly large task, and you are not the
first one to volunteer to do this (previous attempts failed because
people bit more than they could chew).
This is why I am wary of this being the first CLISP project you attempt.

> b) this could be seen as a project (with further details I'd work out
> then) suitable for GSoC.

Let us start small - or, at least, smaller.

Specifically, make a release.
This will require you to learn the current CLISP infrastructure which
will be absolutely critical when you move on to the next step of
cleaning up the code.
This will also allow us, the CLISP maintainers, to pass on our expertise
to you and to monitor and mentor your progress.
It is also a finite task with a clear and visible deliverable.

Currently, CLISP hg tip is in a bad state: an import of gnulib packages
resulted in a code base which probably works only on Linux.

Specifically, what you would need to do is

1. update all gnulib imports

2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
   Windows uses Handles)

3. make sure that CLISP builds and passes self-tests on as many
   platforms as possible, but at least
   A. Linux (fedora, ubuntu &c)
   B. Windows (whatever the most common version is now, with and without
      cygwin)
   C. Mac OS X
   D. *BSD (FreeBSD, OpenBSD, NetBSD)

4. Make an official source and binary release.

I think the above is a good GSoC project.

Bruno, WDYT?

> I'm asking these - probably too detailed - questions on this list here
> instead of on one of the CLISP mailing lists because they seem, as already
> pointed out, to be dead. I was unsure about whether to CC one of the
> maintainers, because it's discouraged in some communities, and so I didn't.
> I didn't CC one of the maintainers because such CCing is discouraged in
> some communities. What's the "correct" way to get in contact here? And: Are
> there any possible mentors for the CLISP project?

Yes, I am prepared to mentor you.
I hope that if you will show some progress, Bruno will chip in too.

> Finally, a very brief info about me: My name is Daniel Jour nee Oertwig,
> I'm currently pursuing my computer science master's degree at University
> Wuerzburg, Germany.
> I'm a self taught programmer, started learning about 14 years ago and have
> been doing all kinds of different things, like some hobby OS kernels,
> programming language ideas, an orbit trajectory calculation tool (I did my
> bachelor's degree in aerospace computer science :) ), C++ code
> transformation tools and a lot more. Thus I've been exposed to a lot of
> technologies, but I'm most proficient with C, C++ and (Common) Lisp and
> operating system concepts.
> I've been working as a teaching assistant for my university's operating
> systems course, teaching mostly stuff related to multithreading (examples
> in Java) and GNU/Linux usage. I'm an active user on stackoverflow.
> I'm 24 years old, married, two kids (and I should note that my family takes
> precedence over anything else, thus I'm probably an a bit "riskier"
> potential GSoC student)
>
> Best regards,
> Daniel Jour

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://think-israel.org http://iris.org.il
http://palestinefacts.org http://americancensorship.org http://dhimmi.org
My plan for 'after work' is to retire.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Hi Sam,

2016-03-21 23:46 GMT+01:00 Sam Steingold <sd...@gn...>:
>
> Hi Daniel,
>
> > * Daniel Jour <qna...@tz...> [2016-03-21 15:07:02 +0000]:
> >
> > the CLISP project seems to be "not that active", judging from the
mailing
> > list / bug tracker / repository activities. The latest release is 6
years
> > old.
>
> Alas, this is, indeed, the case.
>
> > Looking at the code base, I think this low activity stems at least in
part
> > from the IMO relatively bad condition it is in. Most of it seems to be
> > quite old, there are (now talking of the C parts) parts that invoke
> > undefined behavior (general pointer comparisons, integer overflows, out
of
> > bounds indexing) and some other odd things (the whole additional
> > preprocessing, pre C99 code, indexing from 1 instead of 0, apparently no
> > internal interfaces / abstraction, probably more).
> >
> > I'd like to change that. In order to do so I'd like to "modernize" the
code
> > base, starting with the C code. That means among other things
converting to
> > standard compliant code (C99 probably), splitting code into smaller
units,
> > establishing internal interfaces.
> >
> > Before putting (a lot) more work into this I'd like to know whether
> > a) there's is a general interest to make such - probably huge - changes
to
> > the code base, and whether
>
> Yes, there is at least some interest in modernizing CLISP code base.
> Note, however, that this is a fairly large task, and you are not the
> first one to volunteer to do this (previous attempts failed because
> people bit more than they could chew).
> This is why I am wary of this being the first CLISP project you attempt.

That's great. I agree, modernizing the complete code base is a task that's
too huge to attempt as a first CLISP project. What I hoped for was that it's
ok if, picking a random example, readtable related code got extracted from
src/io.d, i.e. it's ok when runtime code is refactored to some degree.

> > b) this could be seen as a project (with further details I'd work out
> > then) suitable for GSoC.
>
> Let us start small - or, at least, smaller.
>
> Specifically, make a release.

Sounds good.

> This will require you to learn the current CLISP infrastructure which
> will be absolutely critical when you move on to the next step of
> cleaning up the code.
> This will also allow us, the CLISP maintainers, to pass on our expertise
> to you and to monitor and mentor your progress.
> It is also a finite task with a clear and visible deliverable.

I think I have already a rough understanding of the infrastructure, in the
sense of "where is what" and how it fits together. (Apart from a lot of
code I read the complete implementation notes and was searching
through the clisp-devel list for info).
Though there's still a lot I don't understand :)

> Currently, CLISP hg tip is in a bad state: an import of gnulib packages
> resulted in a code base which probably works only on Linux.
>
> Specifically, what you would need to do is
>
> 1. update all gnulib imports

To what extend is CLISP using gnulib atm? I've found the
gnulib-imported target in Makefile.devel (as well as related stuff), but am
unsure how far the actual src/* files make use of it. As far as I can tell
only
src/lispbibl.d, src/_clisp.c and src/glmalloc.c seem to use it (only taking
the
runtime into approach, not any modules).

I didn't know about the patches section of sf.net until Ken Brown brought
it up (https://sf.net/p/clisp/mailman/message/34957653/).

> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>    Windows uses Handles)

Can you provide a bit more background on that? Is this related to
http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
thus a general gnulib issue? Or does this only affect CLISP?

> 3. make sure that CLISP builds and passes self-tests on as many
>    platforms as possible, but at least
>    A. Linux (fedora, ubuntu &c)
>    B. Windows (whatever the most common version is now, with and without
>       cygwin)
>    C. Mac OS X
>    D. *BSD (FreeBSD, OpenBSD, NetBSD)

I should be able to get access to all of these:
I'm working on a Linux machine, NixOS distribution (if it compiles here
it'll
probably compile one every Linux, given correct dependencies) and have
access to Debian machines at my university. I have NetBSD and FreeBSD
qemu images available, and can probably get access to both Windows VMs
as well as "real machines" through my university. I can probably also get
access to Mac computers (there's a lab / pool at the university).

How "extensive" are these self-tests?

> 4. Make an official source and binary release.
>
> I think the above is a good GSoC project.

I guess I should get a proposal ready ASAP then.

> Bruno, WDYT?
>
> > I'm asking these - probably too detailed - questions on this list here
> > instead of on one of the CLISP mailing lists because they seem, as
already
> > pointed out, to be dead. I was unsure about whether to CC one of the
> > maintainers, because it's discouraged in some communities, and so I
didn't.
> > I didn't CC one of the maintainers because such CCing is discouraged in
> > some communities. What's the "correct" way to get in contact here? And:
Are
> > there any possible mentors for the CLISP project?
>
> Yes, I am prepared to mentor you.
> I hope that if you will show some progress, Bruno will chip in too.

Thank you a lot. I'll give my best to get a suitable proposal ready
until the deadline.

Daniel

> > Finally, a very brief info about me: My name is Daniel Jour nee Oertwig,
> > I'm currently pursuing my computer science master's degree at University
> > Wuerzburg, Germany.
> > I'm a self taught programmer, started learning about 14 years ago and
have
> > been doing all kinds of different things, like some hobby OS kernels,
> > programming language ideas, an orbit trajectory calculation tool (I did
my
> > bachelor's degree in aerospace computer science :) ), C++ code
> > transformation tools and a lot more. Thus I've been exposed to a lot of
> > technologies, but I'm most proficient with C, C++ and (Common) Lisp and
> > operating system concepts.
> > I've been working as a teaching assistant for my university's operating
> > systems course, teaching mostly stuff related to multithreading
(examples
> > in Java) and GNU/Linux usage. I'm an active user on stackoverflow.
> > I'm 24 years old, married, two kids (and I should note that my family
takes
> > precedence over anything else, thus I'm probably an a bit "riskier"
> > potential GSoC student)
> >
> > Best regards,
> > Daniel Jour
>
> --
> Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X
11.0.11702000
> http://www.childpsy.net/ http://think-israel.org http://iris.org.il
> http://palestinefacts.org http://americancensorship.org http://dhimmi.org
> My plan for 'after work' is to retire.

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Daniel,

Congratulations, too, for obtaining a GSOC!

> I'd like to change that. In order to do so I'd like to "modernize" the code
> base, starting with the C code. That means [...] splitting code into smaller units,

Speaking about lispbibl.d, I'm not sure it's too big, in absolute terms. I've learned
that the "best" file size very much depends on how we use a file, i.e. the tools
we use to manipulate it.

Back in history, approx. 25 years ago, when I first compiled CLISP on a 3MB(?) Amiga,
opening the huge - by old standards - 500kB lispbibl.d took at least once second,
just to load. Searching through 500kB was not instantaneous. Fast forward to the
present, searching in a 1MB file shows no lag.
However, editors are still an order of magnitude better at handling single files
than at working with multiple, related files. How would you specify a search in
only include files, excluding code files, e.g. record.d?

So having everything in once place still has benefits, today.

If you look closely, you'll still find comments like "ARRBIBL for ARRAY.D". Lispbibl.d
back then was a concatenation of those more specific include files. BIBL means
'Bibliothek', i.e. library.
We found out that, due to memory fragmentation issues, it was more reliable to
compile all C files using the one giant include file than the individual pieces.
That's no more an issue.

Last but not least, like it or not, trying to create a proper hierarchy of
tiny ordered includes costs a lot of effort and time. Unsurprisingly, all languages
have something to use forward references. I don't know if it's worth the effort
to change that.

Kind regards,
	Jörg Höhle

Re: clisp - general interest in code cleanup

[<don...@is...>]

Daniel Jour writes:
 > > > Looking at the code base, I think this low activity stems 
My interpretation has been that the low activity is largely due to the
code being GOOD - although I've found things I'd like to change, these
have been easy enough to work around to make the cost/benefit favor
leaving them.

 > I didn't know about the patches section of sf.net until Ken Brown brought
 > it up (https://sf.net/p/clisp/mailman/message/34957653/).

It's news to me too.  In fact I only recently found out that the hg
I had been using for years was not the current one.
Using an old hg makes the activity seem even lower!
I'm still doing several nightly builds to check that any changes don't
break either clisp builds or builds of my own software on top of clisp.

Over the years I've suggested a few small changes on the mailing
list that never made it as far as patches, partly cause I didn't even
know that patches were being collected.  
I'd appreciate if some of those could be included when the patches are
incorporated.

 > > 3. make sure that CLISP builds and passes self-tests on as many
 > >    platforms as possible, but at least
 > >    A. Linux (fedora, ubuntu &c)
 > >    B. Windows (whatever the most common version is now, with and without
 > >       cygwin)
 > >    C. Mac OS X
 > >    D. *BSD (FreeBSD, OpenBSD, NetBSD)

Just wondering, how much interest is there in different linux versions,
and especially old ones?  Some of my nightly test builds are on
Fedora Core release 4 (2.6.17-1.2142_FC4).

Also how much interest is there in different build configurations?
One of the configurations on that old machine,
 ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock build-mt
ends up with
 ./clisp-link: line 97:   538 Segmentation fault      "$@"
(and has for many years now)
This one builds, though:
 ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock --with-dynamic-modules=no build-mt-no

 > How "extensive" are these self-tests?
You'll see.
I consider them to be very extensive.
Sometimes they begin to feel interminable.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Don Cohen <qba...@vf...> [2016-03-23 11:38:39 +0000]:
>
> Also how much interest is there in different build configurations?
> One of the configurations on that old machine,
>  ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock build-mt
> ends up with
>  ./clisp-link: line 97:   538 Segmentation fault      "$@"
> (and has for many years now)
> This one builds, though:
>  ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock --with-dynamic-modules=no build-mt-no

both dynamic modules and threads are important features.
(threads are still experimental though.)
the segmentation fault above is certainly a bug, and should be fixed.
http://www.cygwin.com/acronyms/#PTC

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://mideasttruth.com http://honestreporting.com
http://jihadwatch.org http://islamexposedonline.com http://camera.org
God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-03-23 08:41:00 +0000]:
>
> That's great. I agree, modernizing the complete code base is a task that's
> too huge to attempt as a first CLISP project. What I hoped for was that it's
> ok if, picking a random example, readtable related code got extracted from
> src/io.d, i.e. it's ok when runtime code is refactored to some degree.

this approach does not scale.

>> 1. update all gnulib imports
>
> To what extend is CLISP using gnulib atm? I've found the
> gnulib-imported target in Makefile.devel (as well as related stuff),
> but am unsure how far the actual src/* files make use of it. As far as
> I can tell only src/lispbibl.d, src/_clisp.c and src/glmalloc.c seem
> to use it (only taking the runtime into approach, not any modules).

all files include lispbibl.d, so this means that all files use them. :-)

>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>    Windows uses Handles)
>
> Can you provide a bit more background on that? Is this related to
> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
> thus a general gnulib issue? Or does this only affect CLISP?

this is a general property of gnulib.
clisp used to work around this, but the latest import of gnulib files (5
years ago?) broke it.
IIRC, clisp no longer builds on windows.

> How "extensive" are these self-tests?

there are a lot of them and they cover most of the code.
(modules have their own tests).

> I guess I should get a proposal ready ASAP then.

please do!


-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
http://honestreporting.com http://truepeace.org http://think-israel.org
God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Hi Don,

Don Cohen writes:
>
> Daniel Jour writes:
>  > > > Looking at the code base, I think this low activity stems
> My interpretation has been that the low activity is largely due to the
> code being GOOD - although I've found things I'd like to change, these
> have been easy enough to work around to make the cost/benefit favor
> leaving them.

Please don't get me wrong: The code is good. It's implementing good logic,
making mostly sensible use of the available tools of the C language (I'm
focusing on the runtime implementation). But:

 * The files are huge and contain way too much. Related stuff, but in
principle stuff that makes up separate parts, and could and should be
kept separate.

 * Many functions are also huge, and contain really a lot sections that
are subject to conditional compilation, depending on the target platform
and the selected configuration.

The implication is that in order to make well informed changes to one
part of the code, you also need to know how it interacts with the rest
of the code. If there would be a clean interface then this is not a problem,
because the interface should clearly describe these interactions.

Assume a hobbyist user found a bug in the parser. She's presented with
a 10k+ lines src/io.d which includes (the result of preprocessing) a 17k+
lines src/lispbibl.d. That's a lot of code to even quickly skim over, in order
to make sure not to introduce any unwanted side effects. The
implementation notes help a lot, but provide more of a high level overview,
not so much the interactions between the various code parts.

I think this difficulty to get working with the code is demonstrated by the
low development activity, the age and number of open bugs and the low
number of feature requests (general tail call optimization?).

 * There seems to be a lot of code determining the "available features"
and the "configuration" of the target machine which should be, to at
least some extend, handled by the C library respectively gnulib.

 * Undefined behavior, probably mostly in spvw* and also sort.d:
Stuff like comparing (less, greater) pointers, out of bounds indexing
and some cast just aren't legal C, and can lead to all sorts of nasty bugs,
especially when using an optimizing compiler. I'd guess that's the reason
that (e.g.) my distributions build instructions pass in CFLAGS=-O0.

There's probably more in the direction of general code readability. (A
fun thing to do is the following: Collect the names of all local variables
per function of a source file and the names of the functions. Then shuffle
and try to figure out which set of local variables is part of which function.)

I hope the above didn't make a harsh or arrogant impression, it's basically
how I start when e.g. reviewing code on stackoverflow, and based on the
experiences I made in the last 14 years. I don't assume it to be the "only
way", and I'm completely open to revising my view :) After all, I've never
received any mentoring or formal teaching in programming (I'm not
counting my university ... I can tell with certainty now that universities are
not places that teach "how to code").

> Over the years I've suggested a few small changes on the mailing
> list that never made it as far as patches, partly cause I didn't even
> know that patches were being collected.
> I'd appreciate if some of those could be included when the patches are
> incorporated.

I will search for them in the mailing lists. The more input, the better, IMO.
SF doesn't support searching for names, though, and the search provided
by gmane seems to be working, but "open ended", so it's probable that
I miss some posts. Do you have any "record" of your suggestions?


> Just wondering, how much interest is there in different linux versions,
> and especially old ones?  Some of my nightly test builds are on
> Fedora Core release 4 (2.6.17-1.2142_FC4).

AFAIK the "core" of CLISP doesn't have much dependencies. libsigsegv,
libffi and libffcall (and perhaps readline) are listed as dependencies in
my package description, ldd only shows glibc and pthreads. Assuming
proper use of gnulib CLISP should probably be able to run on most of the
platforms supported by gnulib, which would also include older linux
versions.

CLISP is also known for its portability (and availability, IIRC it was part of
the default installation of debian in the past. At least it's on every machine
in my universities computer pools, and AFAIK the system administrator
hasn't installed it on purpose) so IMO it would be good if a new release
wouldn't break this backwards compability.

But, being completely new here, that's just me guessing :)


>  > How "extensive" are these self-tests?
> You'll see.
> I consider them to be very extensive.
> Sometimes they begin to feel interminable.

Yes, I had the time to read over some of them now. That's basically a huge
suite of regression tests, and that's awesome :)

Daniel

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Daniel Jour wrote (in March):
>* Undefined behavior, probably mostly in spvw* and also sort.d:
>Stuff like comparing (less, greater) pointers, out of bounds indexing and
>some cast just aren't legal C, and can lead to all sorts of nasty bugs,
>especially when using an optimizing compiler. I'd guess that's the reason
>that (e.g.) my distributions build instructions pass in CFLAGS=-O0.

-O0?!? That feels like a slap in the face. Decades ago, we spent countless
hours looking at assembly output and optimizing this and that, measuring
time. We felt that even small performance improvements (i.e. ~2%) were
worth it. That's the origin of the dotimespL() macros. We were very proud
when something produced a 5% improvement. Bruno regularly scrutinized the
bytecode interpreter loop, which is the #1 hot spot.

IIRC, parts of the old libffcall would not even work with -O0, because it
relied on some stack optimizations by GCC, like -fomit-frame-pointer.
(Nowadays, I don't know whether libffcall still makes use of the asm
 files, or compiles even the lowest-level stuff from C).

-O0 output is so pitiful to look at.

Regarding the recent changes in C compilers, I believe it would be
preferable to use, like several other projects, e.g. the Linux kernel or
PostgreSQL, options like -fno-strict-overflow (or -fwrapv),
-fno-delete-null-pointer-checks, -fno-strict-aliasing and whatever else
in order to use optimizations yet behave like all C compilers used
to behave during the last decades. Later, one can add additional casts,
wrapped in yet more macros...;-) to compare pointers of different kinds.

Sam Steingold wrote:
> I see no problem in relying on external libraries.
Nowadays. Initially, Bruno reported that lots of libraries on the many UNIX
systems that CLISP ran on had lots of different bugs, and he was not amused.
As a result, when I did the Amiga port, he advised that we avoid any
lib, thus Amiga-CLISP used nothing but (the equivalent of) syscalls, no
wrappers, and I even wrote the C startup code.
On UNIX, CLISP started using configure.

Regards,
	Jörg Höhle

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Hi Sam,

Sam Steingold writes:
> this approach does not scale.

Can you elaborate on that a bit? As I already wrote to Don, it's just
the way I'm used to, based on previous experiences, but I'm completely
open to learn :) (I'm completely self-taught and have never received
any mentoring or - real, useful - teaching in "programming").


> all files include lispbibl.d, so this means that all files use them. :-)

Hehe, what I meant was more like: There seems to be versions of
alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
that could be also provided by gnulib. That's what's confusing me: Are
these separate versions on purpose, or just not yet replaced by the
Gnulib ones?

>>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>>    Windows uses Handles)
>>
>> Can you provide a bit more background on that? Is this related to
>> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
>> thus a general gnulib issue? Or does this only affect CLISP?
>
> this is a general property of gnulib.
> clisp used to work around this, but the latest import of gnulib files (5
> years ago?) broke it.
> IIRC, clisp no longer builds on windows.

I think it builds on cygwin with Ken's patches, but not "native" windows.
I'll try to get a windows machine / vm up in order to check this.

What's the general course of action here? Fix it in gnulib (as far as that's
possible, eventually using it's local modules feature) or work around "here"?

What priority do non cygwin builds on windows have? Targeting mingw or
the microsoft compilers?


>> How "extensive" are these self-tests?
>
> there are a lot of them and they cover most of the code.
> (modules have their own tests).

I found them, they're awesome.


>> I guess I should get a proposal ready ASAP then.
>
> please do!

Work is in progress, I have rough version that's ready for sharing soon.
Due to the limited time left I add some notes here to start with:

-- Title: CLISP - Make a release (finally)
-- Summary:
CLISP has seen its last release in 2010, almost 6 years ago.
Since then there have been a lot of changes, partly implemented features
and bug fixes. In combination with an import of Gnulib modules this
resulted in a code base that is no longer portable and not suitable for a
release. The purpose of this project is to fix the portability issues and to
cleanly include suitable Gnulib modules, allowing a new release with the
improvements of the past years to be published.

--Raw notes regarding plan:
* Get a complete picture of the code, especially what's broken on non linux
platforms.
* Get rid of (probably? almost) everything that can be provided by a suitable
gnulib module, using the most up to date version of gnulib (replace one part,
then run self tests, update doc, repeat). Potentially extending self tests.
* Fix gnulib on Windows. At least "enough" for CLISP to work.
* Mid-term: core CLISP builds and passes self tests on Linux, BSD, Mac and
Windows.
* update CLISP modules
* include useful (not yet used) patches and changes. It might be worthwhile to
think about holding back (possibly reverting) some half-finished changes here,
keeping them for a next release until they're finished (multithreading? jit?)
* general clean up of code and implementation notes ("polishing for the
release", so to say)
* Finally: release.

-- Skills I already “have”: C and Lisp, knowledge of memory management and
garbage collection as well as general VM/bytecode design and compiler
building. GNU make, Shell programming, general Linux competence.

-- Things I need to learn: Proper usage (as a developer) of the autotools and
the GNU build system, “hands on experience” regarding release process(es),
probably understanding CLISP's VM in depth will help, and finally how to
achieve portability to a lot of platforms. Update of my knowledge of windows
programming (it's been a while).

Sam, do you think that I should include references to previous work? Most
helpful is probably my stackoverflow.com account .. since my github repositories
... ehm .. aren't exactly neat and tidy and contain only a small
(largely outdated)
fraction of the stuff I've been doing all the years :)

Regarding communication: I'm best reached via email, but starting early-mid
April irc becomes an option, too, though I'm constrained regarding time of day.
This might be an issue if you'd like (very) frequent real time communication.
There's not much planned during the coding (and bonding) time span, apart
from a possible holiday of probably a week early August. My exams are at the
end of July, though that won't interfere (I'm thankfully learning fast
and was able
to complete this semesters exams with top scores without being present
or investing
much time during the semester. This has been the same for the last semesters, so
I'm very sure that this will work.)

Do I need to explicitly mention you in the proposal, or is this GNU
internally sorted out?

Looking forward to some feedback (if still possible, I know that only a bit more
than 16 hours left isn't exactly "much time")

Daniel

> --
> Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
> http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
> http://honestreporting.com http://truepeace.org http://think-israel.org
> God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

I hope this is now "close enough" to a proposal in order to be reviewed:
https://docs.google.com/document/d/1VEwlAhA-YNEYiBHzz4jWS9BnRMdxgwF71ihqRdP58uY/edit?usp=sharing

Daniel

2016-03-25 3:32 GMT+01:00 Daniel Jour <dan...@gm...>:
> Hi Sam,
>
> Sam Steingold writes:
>> this approach does not scale.
>
> Can you elaborate on that a bit? As I already wrote to Don, it's just
> the way I'm used to, based on previous experiences, but I'm completely
> open to learn :) (I'm completely self-taught and have never received
> any mentoring or - real, useful - teaching in "programming").
>
>
>> all files include lispbibl.d, so this means that all files use them. :-)
>
> Hehe, what I meant was more like: There seems to be versions of
> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
> that could be also provided by gnulib. That's what's confusing me: Are
> these separate versions on purpose, or just not yet replaced by the
> Gnulib ones?
>
>>>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>>>    Windows uses Handles)
>>>
>>> Can you provide a bit more background on that? Is this related to
>>> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
>>> thus a general gnulib issue? Or does this only affect CLISP?
>>
>> this is a general property of gnulib.
>> clisp used to work around this, but the latest import of gnulib files (5
>> years ago?) broke it.
>> IIRC, clisp no longer builds on windows.
>
> I think it builds on cygwin with Ken's patches, but not "native" windows.
> I'll try to get a windows machine / vm up in order to check this.
>
> What's the general course of action here? Fix it in gnulib (as far as that's
> possible, eventually using it's local modules feature) or work around "here"?
>
> What priority do non cygwin builds on windows have? Targeting mingw or
> the microsoft compilers?
>
>
>>> How "extensive" are these self-tests?
>>
>> there are a lot of them and they cover most of the code.
>> (modules have their own tests).
>
> I found them, they're awesome.
>
>
>>> I guess I should get a proposal ready ASAP then.
>>
>> please do!
>
> Work is in progress, I have rough version that's ready for sharing soon.
> Due to the limited time left I add some notes here to start with:
>
> -- Title: CLISP - Make a release (finally)
> -- Summary:
> CLISP has seen its last release in 2010, almost 6 years ago.
> Since then there have been a lot of changes, partly implemented features
> and bug fixes. In combination with an import of Gnulib modules this
> resulted in a code base that is no longer portable and not suitable for a
> release. The purpose of this project is to fix the portability issues and to
> cleanly include suitable Gnulib modules, allowing a new release with the
> improvements of the past years to be published.
>
> --Raw notes regarding plan:
> * Get a complete picture of the code, especially what's broken on non linux
> platforms.
> * Get rid of (probably? almost) everything that can be provided by a suitable
> gnulib module, using the most up to date version of gnulib (replace one part,
> then run self tests, update doc, repeat). Potentially extending self tests.
> * Fix gnulib on Windows. At least "enough" for CLISP to work.
> * Mid-term: core CLISP builds and passes self tests on Linux, BSD, Mac and
> Windows.
> * update CLISP modules
> * include useful (not yet used) patches and changes. It might be worthwhile to
> think about holding back (possibly reverting) some half-finished changes here,
> keeping them for a next release until they're finished (multithreading? jit?)
> * general clean up of code and implementation notes ("polishing for the
> release", so to say)
> * Finally: release.
>
> -- Skills I already “have”: C and Lisp, knowledge of memory management and
> garbage collection as well as general VM/bytecode design and compiler
> building. GNU make, Shell programming, general Linux competence.
>
> -- Things I need to learn: Proper usage (as a developer) of the autotools and
> the GNU build system, “hands on experience” regarding release process(es),
> probably understanding CLISP's VM in depth will help, and finally how to
> achieve portability to a lot of platforms. Update of my knowledge of windows
> programming (it's been a while).
>
> Sam, do you think that I should include references to previous work? Most
> helpful is probably my stackoverflow.com account .. since my github repositories
> ... ehm .. aren't exactly neat and tidy and contain only a small
> (largely outdated)
> fraction of the stuff I've been doing all the years :)
>
> Regarding communication: I'm best reached via email, but starting early-mid
> April irc becomes an option, too, though I'm constrained regarding time of day.
> This might be an issue if you'd like (very) frequent real time communication.
> There's not much planned during the coding (and bonding) time span, apart
> from a possible holiday of probably a week early August. My exams are at the
> end of July, though that won't interfere (I'm thankfully learning fast
> and was able
> to complete this semesters exams with top scores without being present
> or investing
> much time during the semester. This has been the same for the last semesters, so
> I'm very sure that this will work.)
>
> Do I need to explicitly mention you in the proposal, or is this GNU
> internally sorted out?
>
> Looking forward to some feedback (if still possible, I know that only a bit more
> than 16 hours left isn't exactly "much time")
>
> Daniel
>
>> --
>> Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
>> http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
>> http://honestreporting.com http://truepeace.org http://think-israel.org
>> God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-03-25 04:59:01 +0100]:
>
> I hope this is now "close enough" to a proposal in order to be reviewed:
> https://docs.google.com/document/d/1VEwlAhA-YNEYiBHzz4jWS9BnRMdxgwF71ihqRdP58uY/edit?usp=sharing

I added a couple minor comments, otherwise lgtm.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://honestreporting.com http://iris.org.il
http://memri.org http://dhimmi.org http://camera.org
C combines the power of assembler with the portability of assembler.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-03-25 03:32:51 +0100]:
>
> Sam Steingold writes:
>> this approach does not scale.
>
> Can you elaborate on that a bit?

The approach of you refactoring many huge files and us reviewing your
changes does not scale because none of us have time for that.

>> all files include lispbibl.d, so this means that all files use them. :-)
>
> Hehe, what I meant was more like: There seems to be versions of
> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
> that could be also provided by gnulib. That's what's confusing me: Are
> these separate versions on purpose, or just not yet replaced by the
> Gnulib ones?

CLISP has a lot of code which is now available elsewhere because much of
CLISP was written 20+ years ago.

I see no problem in relying on external libraries.
I am not sure that gnulib provides avl trees though.

> What's the general course of action here? Fix it in gnulib (as far as
> that's possible, eventually using it's local modules feature) or work
> around "here"?

I suggest that you discuss this on the gnulib mailing list.
I have no idea what the gnulib people will recommend.

> What priority do non cygwin builds on windows have? Targeting mingw or
> the microsoft compilers?

Up to you - whatever compiler is easier to use/gives better speed/size
optimization &c.
I guess mingw is easier because it's the same gcc.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://truepeace.org http://jihadwatch.org
http://dhimmi.org http://think-israel.org http://honestreporting.com
War has never solved anything - except for ending Slavery, Nazism, Communism.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Sam Steingold writes:
> The approach of you refactoring many huge files and us reviewing your
> changes does not scale because none of us have time for that.

Oh, I see! That indeed wouldn't make much sense now.

>>> all files include lispbibl.d, so this means that all files use them. :-)
>>
>> Hehe, what I meant was more like: There seems to be versions of
>> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
>> that could be also provided by gnulib. That's what's confusing me: Are
>> these separate versions on purpose, or just not yet replaced by the
>> Gnulib ones?
>
> CLISP has a lot of code which is now available elsewhere because much of
> CLISP was written 20+ years ago.
>
> I see no problem in relying on external libraries.
> I am not sure that gnulib provides avl trees though.

Good. It does, there are three implementations, written by Bruno Haible, so
I'd guess they're probably very similar or even the same as the CLISP
implementation:
https://www.gnu.org/software/gnulib/MODULES.html#ansic_ext_container

>> What's the general course of action here? Fix it in gnulib (as far as
>> that's possible, eventually using it's local modules feature) or work
>> around "here"?
>
> I suggest that you discuss this on the gnulib mailing list.
> I have no idea what the gnulib people will recommend.

Ok, I'll do that (searching the list for preliminary information now).

> I added a couple minor comments, otherwise lgtm.

Thank you :)

Fwd: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

(Resend to list, sorry for the duplication)

> Congratulations, too, for obtaining a GSOC!

Thank you very much!

> Speaking about lispbibl.d, I'm not sure it's too big, in absolute terms.
> I've learned that the "best" file size very much depends on how we use
> a file, i.e. the tools we use to manipulate it.

Definitely. I'm not so much concerned about the actual size (lines of
code, space on disk), but rather the amount of unrelated "topics" that are
put together. Example: Startup code, C string manipulation and argument
parsing live together with memory management in src/spvw.d. This is
especially
concerning if the "topics" are intermingled.

If the only "use" of the code is when we compile the final executable, then
having all in one (or some) huge files is not so much an issue (though one
looses the ability to have functions / types / macros with the same name,
in case that's needed). But if you only want to "use" a small part of the
code, e.g. in order to test that part, that gets pretty difficult.

> However, editors are still an order of magnitude better at handling single
> files than at working with multiple, related files. How would you specify
> a search in only include files, excluding code files, e.g. record.d?

I don't think so. Not even talking about what "complex" IDEs can do, one can
always fall back to "the right tool for the job", in this case (for example)
GNU find and grep. (Emacs with projectile makes this available via C-c p s
g ..
which I read as "command project(ile) search (via) grep". There are also
ways to do semantic searches like finding callers, macro expansion points
etc,
but the simple text search has been enough for me ever since.)

To search only in include files, one would only grep in these.
Include files (normally) should have a different filename extension than
implementation files.

Consider this (contrived example):
Suppose you'd like to change the readtable related code to
not (only) work with simple vectors but also with some other (fancy) data
structure. For that you want to find all uses of simple_vector_p in those
parts of the code.

If "topics" are in different files, then one can narrow down searches based
on that, which is a huge win IMO.

> If you look closely, you'll still find comments like "ARRBIBL for
ARRAY.D".
> Lispbibl.d back then was a concatenation of those more specific include
> files. BIBL means 'Bibliothek', i.e. library. We found out that, due to
> memory fragmentation issues, it was more reliable to compile all C files
> using the one giant include file than the individual pieces.
> That's no more an issue.

That must have been tough times :) I've always been working on machines with
(compared to that) plentiful memory available, so that I never had to worry
about not having enough memory to safely run a compiler.

Why didn't you run the concatenation as a build step?


> Last but not least, like it or not, trying to create a proper hierarchy of
> tiny ordered includes costs a lot of effort and time. Unsurprisingly, all
> languages have something to use forward references. I don't know if it's
> worth the effort to change that.

That's true: "Where do I put that?" together with "How do I name that?" are
often surprisingly difficult to answer. Though I don't understand what you
mean with the forward references?

Best,
Daniel

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Daniel Jour wrote:

> Startup code, C string manipulation and argument parsing live together
>with memory management in src/spvw.d. This is especially concerning if the
>"topics" are intermingled.
It's always interesting when new, i.e. unbiased eyes look at old code. To
me, having arg parsing and mm in the file called spvw = Speicherverwaltung
= memory management was logical. This (set of) files contain(s) all the
low-level stuff that help create the Lisp world (of objects) that the
other files all use.

> (Emacs with projectile makes this available via C-c p s g ..
Thanks for the pointer. I should look into some of the "newer" Emacs
packages. I believe most of the commands I use already were in Emacs 18 ;-)


>Why didn't you run the concatenation as a build step?
IIRC, Bruno's Makefile did this. I didn't, because of disk space. A 1MB
file on a 80MB drive was a lot, so I preferred to work with temporary
files, i.e. only keep .d and .o on disk.

Actually, I can't remember. Perhaps for exactly the reasons mentioned
previously: comfort of working with basically one include (lispbibl.d)
and one code (e.g. spvw.d) file, each in one window of my (Emacs),
Bruno's (axe) editor or whatever people used. Emacs' incremental search
is often more effective than any combination of grep I know, even if
you add some context lines with -A or -B (sometimes grep is better,
e.g. to show all matches in tabular form).

Compare this with Linux includes, which are often annoyingly deep. I simply
view this as different design choices. Bruno preferred to e.g. cover all
variations of UNIXisms next to each other in unix.d and spvw.d, others
prefer lots of different files. Each choice makes some operations easy
and others difficult.


>Though I don't understand what you mean with the forward references?
You should first define types, then use them in function declarations;
first define low-level functions, then the callers of these. That's
the typical Pascal pyramidal ordering of program elements. For some
programs and data structures you can't preserve this hierarchical
ordering, you need forward references (IIRC, the Pascal programming
language has them too).

CLISP code is ordered like much this and I fully agree with Bruno who
once told me that it makes reviewing code easier as well as more robust.
I believe the reason is that it avoids backtracking, and the human brain
is particularly bad at backtracking, i.e. forgetting false assumptions
about behavior of code.

When you look at a function and see some call, it helps enormously to
*know* what the function does *exactly* -- because you already reviewed
it, reading from top to bottom through the file -- than to have to guess
from the name or the 10 line header description whether it may cover
this or that corner case, e.g.
 - is the second parameter allowed to be NULL,
 - is the upper bound inclusive or not,
 - what if the array is empty,
 - does it assume that lock X is held,
 - will it release lock X in case of error, etc.


>I'm not really experienced with keeping a changelog. As the CLISP
>changelog seems to be pretty detailed: How do you maintain it?
There's an Emacs mode adding nice colors while editing.

Regards,
	Jörg Höhle

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

>> Startup code, C string manipulation and argument parsing live together
>>with memory management in src/spvw.d. This is especially concerning if the
>>"topics" are intermingled.
> It's always interesting when new, i.e. unbiased eyes look at old code. To
> me, having arg parsing and mm in the file called spvw = Speicherverwaltung
> = memory management was logical. This (set of) files contain(s) all the
> low-level stuff that help create the Lisp world (of objects) that the
> other files all use.

When I read Speicherverwaltung (I'm unsure whether you know: I'm German :) ),
then what comes first to my mind is "malloc" and "free".
Thus, I'm basically expecting some sort of interface where I can request (and
free, normally) pieces of memory that satisfy some requirements (like size,
alignment) and which I can then use to create "objects" in them.
This view is probably heavily influenced by C++, though.

I also always favoured separation of "startup" code and "normal operation"
code. (When working on a OS kernel, that separation helps to get rid of the
former once the system is up)


>>Why didn't you run the concatenation as a build step?
> IIRC, Bruno's Makefile did this. I didn't, because of disk space. A 1MB
> file on a 80MB drive was a lot, so I preferred to work with temporary
> files, i.e. only keep .d and .o on disk.
>
> Actually, I can't remember. Perhaps for exactly the reasons mentioned
> previously: comfort of working with basically one include (lispbibl.d)
> and one code (e.g. spvw.d) file, each in one window of my (Emacs),
> Bruno's (axe) editor or whatever people used. Emacs' incremental search
> is often more effective than any combination of grep I know, even if
> you add some context lines with -A or -B (sometimes grep is better,
> e.g. to show all matches in tabular form).

I always forget how limited memory once was :) While reading further through
CLISP's code I found Emacs' search to be more effective than grepping at times
when I wanted to go to the previous or next occurence, starting from the
current point.


> Compare this with Linux includes, which are often annoyingly deep. I simply
> view this as different design choices. Bruno preferred to e.g. cover all
> variations of UNIXisms next to each other in unix.d and spvw.d, others
> prefer lots of different files. Each choice makes some operations easy
> and others difficult.

Searching through Linux includes can be a real pain, indeed. I'm glad to
be able to experience what it's like working with the other "extreme".


>>Though I don't understand what you mean with the forward references?
> You should first define types, then use them in function declarations;
> first define low-level functions, then the callers of these. That's
> the typical Pascal pyramidal ordering of program elements. For some
> programs and data structures you can't preserve this hierarchical
> ordering, you need forward references (IIRC, the Pascal programming
> language has them too).

So (regarding C) you mean forward declarations of structs and functions,
and you opt against them when possible (by keeping code that references
functions/structures "beneath" them)? That's a good approach IMO, and it
also works well with small files (albeit it's sometimes difficult to break
cyclic dependencies).
The way you describe that reminds me of "bottom up" programming (as
described by Paul Graham).


> CLISP code is ordered like much this and I fully agree with Bruno who
> once told me that it makes reviewing code easier as well as more robust.
> I believe the reason is that it avoids backtracking, and the human brain
> is particularly bad at backtracking, i.e. forgetting false assumptions
> about behavior of code.

Yes, forgetting a false assumption about a piece of code can be hard. OTOH
I found it sometimes to be helpful to not know exactly what some function
does but to first see how it's used: This made it easier to understand the
function itself.


> When you look at a function and see some call, it helps enormously to
> *know* what the function does *exactly* -- because you already reviewed
> it, reading from top to bottom through the file -- than to have to guess
> from the name or the 10 line header description whether it may cover
> this or that corner case, e.g.
>  - is the second parameter allowed to be NULL,
>  - is the upper bound inclusive or not,
>  - what if the array is empty,
>  - does it assume that lock X is held,
>  - will it release lock X in case of error, etc.

Reading from top to bottom through the file ... that reminds me a bit of
literate programming (I tried this only in Haskell). I guess it's a bit
like reading a book: You don't want to have to jump 100 pages ahead in
order to understand the current chapter. Structuring code like that is
IMO good, though it only works if one really starts reading from the top.

The issue is if one does not have the time to start reading from the top,
but must/want to make a change now. If you jump right into spvw.d and
start reading code, then there's a lot that you have to look up (and
eventually "backtrack", if you made a false assumption).

That's why I like "layered" approaches: You put low level code into a
separate file and create an include file that contains only what you're
supposed to be using in "higher" layers together with a extensive summary
of how these functions and structures behave and are to be used.
(Though that's not an argument for separate files ... all of this could
of course live in the same file; I think it's the "summary" that's the
critical thing which is missing for some parts of the CLISP code)

Re: clisp - general interest in code cleanup

[Pascal J. B. <pj...@in...>]

Daniel Jour <dan...@gm...> writes:

> The issue is if one does not have the time to start reading from the top,
> but must/want to make a change now. If you jump right into spvw.d and
> start reading code, then there's a lot that you have to look up (and
> eventually "backtrack", if you made a false assumption).
>
> That's why I like "layered" approaches: You put low level code into a
> separate file and create an include file that contains only what you're
> supposed to be using in "higher" layers together with a extensive summary
> of how these functions and structures behave and are to be used.

I would not want to detract you from your clisp maintainance, but for a
later project, we could imagine an emacs mode where each function (or
toplevel form) is presented in its own buffer, with navigation buttons
to the callers and callees and other references to the function.  emacs
would then manage itself the storing of the toplevel forms in one or
more files files, you wouldn't have to care about it.  Something like
the Smalltalk code browser: there's no notion of file then.

> (Though that's not an argument for separate files ... all of this could
> of course live in the same file; I think it's the "summary" that's the
> critical thing which is missing for some parts of the CLISP code)

Don't complain, at least you have the specifications of this program!
:-)


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Pascal J. Bourguignon wrote:

> [...] for a later project, we could imagine an emacs mode where
> each function (or toplevel form) is presented in its own buffer,
> with navigation buttons to the callers and callees and other
> references to the function.  

This remembers me of the Visual Basic Editor in MS-Word. It always
felt like an aberration to me (and I've heard others express similar
feelings), because of total lack of orientation, never knowing
whether there's perhaps another method, callback or event handler
that's needed to understand the code, hidden somewhere else, only
MS gurus know where!

Compare with a bottom up file: you have all in code in front of you.
Print it on paper for even better reading (I haven't done so for
long, dreaming of a paperless office, but you'll hear some well-known
SW-gurus really recommend this).

Perhaps I'm old-fashioned. The TV & monitors I've bought have grown
bigger each and every time. Other people seem to favour the opposite:
they work with Eclipse or a similar IDE, and with every release, it
feels like the code window gets smaller and smaller, because the
information windows to the left, to the right, to the top and to the
bottom grow, yielding a very narrow (let's say "focused" to express
it in a positive way) view on what's actually important. As a result,
those days, programmers view code in a tinier window than when I
started computing on a 9" screen over 30 years ago.
That's no progress to me!


> Something like the Smalltalk
> code browser: there's no notion of file then.

It works nicely for Smalltalk, because unlike C or Common Lisp, the
semantics of a method body does not depend on a myriad of definitions
that appears in lexical order before it in the source file.
 - No macros nor any variations to the syntax;
 - Nothing like operator precedence (except unary operators);
 - class and instance variables are one window away.
Hence IMHO, any parallel is not applicable.

In summary, the 'function at a time' might work with programs with
strong locality, e.g. tree structured. But not when you need to
implement a collection of event handlers. Those hide the flow and
ordering of messages and calls -- in Smalltalk too.

Regards,
	Jörg Höhle

Re: clisp - general interest in code cleanup

[Pascal J. B. <pj...@in...>]

<Joe...@t-...> writes:

> it in a positive way) view on what's actually important. As a result,
> those days, programmers view code in a tinier window than when I
> started computing on a 9" screen over 30 years ago.
> That's no progress to me!

LOL. Indeed.

>> Something like the Smalltalk
>> code browser: there's no notion of file then.
>
> It works nicely for Smalltalk, because unlike C or Common Lisp, the
> semantics of a method body does not depend on a myriad of definitions
> that appears in lexical order before it in the source file.
>  - No macros nor any variations to the syntax;
>  - Nothing like operator precedence (except unary operators);
>  - class and instance variables are one window away.
> Hence IMHO, any parallel is not applicable.

Reader macros can be treated as a dependency (at the form level).

But you're right that a requirement for an IDE nowadays is to include an
execution environment where the source being edited is also compiled and
loaded.  In modern Java IDE and C/C++ (with clang) IDE, they compile the
source at least.  In lisp we'd have to add the run-time environment, to
be able to identify correctly more elements, notably if we don't have a
global static analyser.  This is what slime does.

Granted it's possible to write lisp programs that will fight even this
step, but remember that macros and reader macros should be idempotent,
so there's already some provision in the standard to help IDE there.
Also, with the promotion of ASDF, package and system set up has become
more declarative (rather than procedural); modern lisp code is more
compatible with an IDE treatment.


> In summary, the 'function at a time' might work with programs with
> strong locality, e.g. tree structured. But not when you need to
> implement a collection of event handlers. Those hide the flow and
> ordering of messages and calls -- in Smalltalk too.

Indeed, I had such an example in a recent Android program I had to
maintain; it sent events to communicate between modules (assumedly on
different threads).  This kind of dependency could also be taken into
account by the IDE and it could provide navigation helpers, either
because (as in the case of this Android program) it uses a "standard"
communication mechanis, or because the programmer could declare such
dependencies to the IDE. Notably in lisp, we have the advantage of using
symbols that can be easily identified across modules.  Navigating to
uses of a given symbol could be all that is needed for the basic
feature.


-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk