clisp-devel — closed list for developers and testers

Re: multiple values: value1 treated extra even when not in register

[Daniel J. <dan...@gm...>]

> I suggest that you make a note about this (e.g., by filing a bug or a
> patch) and revisit it after you make the release.

Why not include it in the release? It seems like a minor change with
manageable consequences to me. (I guess understanding this is
very important, so that I can focus on changes that should make it
into the release :) )

Re: multiple values: value1 treated extra even when not in register

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-03 10:03:03 +0200]:
>
>> it might be that the define was not removed by mistake.
>> you can try to figure it out using 'hg blame' and searching through ChangeLog.
>
> It's changeset 93dee6a014b2 from January 2008: You gave global variables (like
> the multiple value space) thread local storage, and changed:
>
>  # Synonyms:
>  #if !defined(value1_register)
> -  #ifndef MULTITHREAD
>      #define value1  mv_space[0]
> -  #else
> -    /* The first value mv_space[0] is moved to the beginning of struct
> -       clisp_thread_t: */
> -    #define value1  (current_thread()->_value1)
> -    #define VALUE1_EXTRA # and thus has to be treated extra every time...
> -  #endif
> +  #define VALUE1_EXTRA /* and thus has to be treated extra every time... */
>  #else
>
> Before that change value1 lived in a dedicated thread structure if
> multithreading was enabled (and there was no register for it). Thus it had
> to be treated extra. Now it's just part of a (thread local) global variable,
> which is addressable and thus does not need to be treated any different.
>
> Though that extra treatment does not lead to unwanted behavior, it's just doing
> more work.

Looks like an oversight then.
When Bruno made releases, he carefully benchmarked everything.
I did not, so this is how this probably slipped through.
I suggest that you make a note about this (e.g., by filing a bug or a
patch) and revisit it after you make the release.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://iris.org.il http://www.memritv.org
http://mideasttruth.com http://www.dhimmitude.org http://palestinefacts.org
Ph.D. stands for "Phony Doctor"  - Isaak Asimov, Ph.D.

Re: multiple values: value1 treated extra even when not in register

[Daniel J. <dan...@gm...>]

> it might be that the define was not removed by mistake.
> you can try to figure it out using 'hg blame' and searching through ChangeLog.

It's changeset 93dee6a014b2 from January 2008: You gave global variables (like
the multiple value space) thread local storage, and changed:

 # Synonyms:
 #if !defined(value1_register)
-  #ifndef MULTITHREAD
     #define value1  mv_space[0]
-  #else
-    /* The first value mv_space[0] is moved to the beginning of struct
-       clisp_thread_t: */
-    #define value1  (current_thread()->_value1)
-    #define VALUE1_EXTRA # and thus has to be treated extra every time...
-  #endif
+  #define VALUE1_EXTRA /* and thus has to be treated extra every time... */
 #else

Before that change value1 lived in a dedicated thread structure if
multithreading was enabled (and there was no register for it). Thus it had
to be treated extra. Now it's just part of a (thread local) global variable,
which is addressable and thus does not need to be treated any different.

Though that extra treatment does not lead to unwanted behavior, it's just doing
more work.

Re: GSoC 2016: proposal accepted!

[Sam S. <sd...@gn...>]

Jörg & Daniel,
Last time I built clisp, it worked.
It is dog-slow, but I have never released clisp without first running
"make check" with DEBUG_GCSAFETY.
Briefly, CLISP GC moves memory, so any allocating function can trigger
GC and thus invalidates all objects.
Running at top-level
$ ./configure CC=g++ --with-debug build-debug-gxx-g
will configure CLISP in directory build-debug-gxx-g enabling various
run-time checks, and CC=g++ enables DEBUG_GCSAFETY (see makemake.in).
Thanks for mentioning this.
Sam


On Mon, May 2, 2016 at 1:35 PM,  <Joe...@t-...> wrote:
> Daniel Jour wrote:
>>The testsuite is really impressive, though it doesn't help much when making
>>little changes. Thus I think about adding some tests that are closer to what
>>they tests.
>
> Sam, how to activate the C++ GC checker that makes use of the maygc annotations?
> Has it been used during later releases or is it bit rot?
> I believe it would be a valuable tool that helps checking low-level C code changes.
>
> If it still works, better not release anything without using it (I mean during
> pre-release tests, not in the released binary).
>
> OTOH, it's presumably not the first thing to try when initially diving through the code.
>
> Regards,
>         Jörg



-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net/>

Re: GSoC 2016: proposal accepted!

[Vladimir T. <vtz...@gm...>]

On Mon, May 2, 2016 at 8:35 PM,  <Joe...@t-...> wrote:
> Sam, how to activate the C++ GC checker that makes use of the maygc annotations?

CC=g++ and configure --with-debug, afair.

> Has it been used during later releases or is it bit rot?

I used and found it valuable while working on multithreading.

BR
 Vlad

Re: GSoC 2016: proposal accepted!

[<Joe...@t-...>]

Daniel Jour wrote:
>The testsuite is really impressive, though it doesn't help much when making
>little changes. Thus I think about adding some tests that are closer to what
>they tests.

Sam, how to activate the C++ GC checker that makes use of the maygc annotations?
Has it been used during later releases or is it bit rot?
I believe it would be a valuable tool that helps checking low-level C code changes.

If it still works, better not release anything without using it (I mean during
pre-release tests, not in the released binary).

OTOH, it's presumably not the first thing to try when initially diving through the code.

Regards,
	Jörg

Re: GSoC 2016: proposal accepted!

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-04-30 23:38:52 +0200]:
>
> The testsuite is really impressive, though it doesn't help much when making
> little changes. Thus I think about adding some tests that are closer to what
> they tests.

the more the merrier.

> I'm not really experienced with keeping a changelog. As the CLISP
> changelog seems to be pretty detailed: How do you maintain it?

Emacs has a good ChangeLog mode.
basically, if you use clisp/emacs/d-mode.el and type M-x
add-change-log-entry RET, Emacs should DTRT.

> There are two accepted projects for libffcall:
> https://summerofcode.withgoogle.com/projects/#5894719238307840
> https://summerofcode.withgoogle.com/projects/#5412443132002304
> Do you know anything about these? As far as I know CLISP is the
> main "user" of that library - shall I try to contact them to
> know "what they're up to"?

Sure, you can talk to them :-)
I think they are doing pretty much what you are doing - fixing up for a
release, merging in various patches &c &c

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://openvotingconsortium.org http://memri.org
http://camera.org http://think-israel.org http://truepeace.org
Don't wait for the rain to end. Learn to dance in the rain!

Re: multiple values: value1 treated extra even when not in register

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-05-01 00:31:13 +0000]:
>
> I was just going over the code regarding multiple values when this
> caught my attention: In both cases, that is if value1_register is
> defined and when it's not, VALUE1_EXTRA is defined. Thus value1 is
> always treated extra

it might be that the define was not removed by mistake.
you can try to figure it out using 'hg blame' and searching through ChangeLog.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://memri.org http://ffii.org
http://palestinefacts.org http://iris.org.il http://americancensorship.org
If you think big enough, you'll never have to do it.

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Hi,

Daniel Jour wrote:

> Startup code, C string manipulation and argument parsing live together
>with memory management in src/spvw.d. This is especially concerning if the
>"topics" are intermingled.
It's always interesting when new, i.e. unbiased eyes look at old code. To
me, having arg parsing and mm in the file called spvw = Speicherverwaltung
= memory management was logical. This (set of) files contain(s) all the
low-level stuff that help create the Lisp world (of objects) that the
other files all use.

> (Emacs with projectile makes this available via C-c p s g ..
Thanks for the pointer. I should look into some of the "newer" Emacs
packages. I believe most of the commands I use already were in Emacs 18 ;-)


>Why didn't you run the concatenation as a build step?
IIRC, Bruno's Makefile did this. I didn't, because of disk space. A 1MB
file on a 80MB drive was a lot, so I preferred to work with temporary
files, i.e. only keep .d and .o on disk.

Actually, I can't remember. Perhaps for exactly the reasons mentioned
previously: comfort of working with basically one include (lispbibl.d)
and one code (e.g. spvw.d) file, each in one window of my (Emacs),
Bruno's (axe) editor or whatever people used. Emacs' incremental search
is often more effective than any combination of grep I know, even if
you add some context lines with -A or -B (sometimes grep is better,
e.g. to show all matches in tabular form).

Compare this with Linux includes, which are often annoyingly deep. I simply
view this as different design choices. Bruno preferred to e.g. cover all
variations of UNIXisms next to each other in unix.d and spvw.d, others
prefer lots of different files. Each choice makes some operations easy
and others difficult.


>Though I don't understand what you mean with the forward references?
You should first define types, then use them in function declarations;
first define low-level functions, then the callers of these. That's
the typical Pascal pyramidal ordering of program elements. For some
programs and data structures you can't preserve this hierarchical
ordering, you need forward references (IIRC, the Pascal programming
language has them too).

CLISP code is ordered like much this and I fully agree with Bruno who
once told me that it makes reviewing code easier as well as more robust.
I believe the reason is that it avoids backtracking, and the human brain
is particularly bad at backtracking, i.e. forgetting false assumptions
about behavior of code.

When you look at a function and see some call, it helps enormously to
*know* what the function does *exactly* -- because you already reviewed
it, reading from top to bottom through the file -- than to have to guess
from the name or the 10 line header description whether it may cover
this or that corner case, e.g.
 - is the second parameter allowed to be NULL,
 - is the upper bound inclusive or not,
 - what if the array is empty,
 - does it assume that lock X is held,
 - will it release lock X in case of error, etc.


>I'm not really experienced with keeping a changelog. As the CLISP
>changelog seems to be pretty detailed: How do you maintain it?
There's an Emacs mode adding nice colors while editing.

Regards,
	Jörg Höhle

multiple values: value1 treated extra even when not in register

[Daniel J. <dan...@gm...>]

I was just going over the code regarding multiple values when this caught
my attention:
In both cases, that is if value1_register is defined and when it's not,
VALUE1_EXTRA is
defined. Thus value1 is always treated extra. This affects:

STACK_to_mv
mv_to_STACK
list_to_mv
do_intern
do_find_symbol

Am I right that this is an unnoticed (thanks to the also working behavior
of the "extra"
treatment) bug?

Results of the tests seems to be not affected in any way (fails at exactly
the same test).

diff -r a6632030649a src/lispbibl.d
--- a/src/lispbibl.d Sat Apr 30 10:06:04 2016 +0200
+++ b/src/lispbibl.d Sun May 01 02:23:18 2016 +0200
@@ -11587,7 +11587,6 @@
 /* Synonyms: */
 #if !defined(value1_register)
     #define value1  mv_space[0]
-  #define VALUE1_EXTRA /* and thus has to be treated extra every time... */
 #else
   /* The first value mv_space[0] is stored permanently in a register: */
   register object value1 __asm__(value1_register);

Fwd: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

(Resend to list, sorry for the duplication)

> Congratulations, too, for obtaining a GSOC!

Thank you very much!

> Speaking about lispbibl.d, I'm not sure it's too big, in absolute terms.
> I've learned that the "best" file size very much depends on how we use
> a file, i.e. the tools we use to manipulate it.

Definitely. I'm not so much concerned about the actual size (lines of
code, space on disk), but rather the amount of unrelated "topics" that are
put together. Example: Startup code, C string manipulation and argument
parsing live together with memory management in src/spvw.d. This is
especially
concerning if the "topics" are intermingled.

If the only "use" of the code is when we compile the final executable, then
having all in one (or some) huge files is not so much an issue (though one
looses the ability to have functions / types / macros with the same name,
in case that's needed). But if you only want to "use" a small part of the
code, e.g. in order to test that part, that gets pretty difficult.

> However, editors are still an order of magnitude better at handling single
> files than at working with multiple, related files. How would you specify
> a search in only include files, excluding code files, e.g. record.d?

I don't think so. Not even talking about what "complex" IDEs can do, one can
always fall back to "the right tool for the job", in this case (for example)
GNU find and grep. (Emacs with projectile makes this available via C-c p s
g ..
which I read as "command project(ile) search (via) grep". There are also
ways to do semantic searches like finding callers, macro expansion points
etc,
but the simple text search has been enough for me ever since.)

To search only in include files, one would only grep in these.
Include files (normally) should have a different filename extension than
implementation files.

Consider this (contrived example):
Suppose you'd like to change the readtable related code to
not (only) work with simple vectors but also with some other (fancy) data
structure. For that you want to find all uses of simple_vector_p in those
parts of the code.

If "topics" are in different files, then one can narrow down searches based
on that, which is a huge win IMO.

> If you look closely, you'll still find comments like "ARRBIBL for
ARRAY.D".
> Lispbibl.d back then was a concatenation of those more specific include
> files. BIBL means 'Bibliothek', i.e. library. We found out that, due to
> memory fragmentation issues, it was more reliable to compile all C files
> using the one giant include file than the individual pieces.
> That's no more an issue.

That must have been tough times :) I've always been working on machines with
(compared to that) plentiful memory available, so that I never had to worry
about not having enough memory to safely run a compiler.

Why didn't you run the concatenation as a build step?


> Last but not least, like it or not, trying to create a proper hierarchy of
> tiny ordered includes costs a lot of effort and time. Unsurprisingly, all
> languages have something to use forward references. I don't know if it's
> worth the effort to change that.

That's true: "Where do I put that?" together with "How do I name that?" are
often surprisingly difficult to answer. Though I don't understand what you
mean with the forward references?

Best,
Daniel

Re: GSoC 2016: proposal accepted!

[Daniel J. <dan...@gm...>]

> CLISP style is more geared towards "compactness" (e.g., only the closing
> brace is ever on its own line, never the opening - unless it is the
> first in a function and thus in the first column; no more than 2 spaces
> for indentation &c).

> I suggest sticking with the current conventions for now.

I stick with the var macro and the CLISP style then, falling back to GNU
where CLISP style is "underspecified".


> Your next step after making the release might be removing varbrace and
> requiring C99 - are there any major compilers which don't support it?

As far as I know all support it at least to the extend that we could use
what's provided by gnulib. I'll investigate more on that after GSOC :)


> Not really.  CLISP is GNU GPL v2+, so all your code is to be released
> under that license; I assume you understand that. :-)
> A simple note in an e-mail that you acknowledge and accept that fact and
> do consent to releasing your code under GPLv2+ is fine.

I understand, acknowledge and accept that all of my code for CLISP is to be
released under GNU GPL v2+.


> If you have no questions for a week, please send e-mail here describing
> your progress.
> I am afraid you will be asking questions more frequently than sending
> progress reports though :-)

That's more than likely :)

I'm currently still working through the code in order to understand better
how the various parts fit together (it's still "community bonding period",
so I guess this is ok). Compiling with -fdump-rtl-expand and using egypt
helped a lot. I'm also having a hard time reproducing some crashes when
running under GDB, which is probably due to lots of places with undefined
behaviour. I'll try to get rid of them soon.

The testsuite is really impressive, though it doesn't help much when making
little changes. Thus I think about adding some tests that are closer to what
they tests.

I'm not really experienced with keeping a changelog. As the CLISP
changelog seems to be pretty detailed: How do you maintain it?

There are two accepted projects for libffcall:
https://summerofcode.withgoogle.com/projects/#5894719238307840
https://summerofcode.withgoogle.com/projects/#5412443132002304
Do you know anything about these? As far as I know CLISP is the
main "user" of that library - shall I try to contact them to
know "what they're up to"?

Re: clisp - general interest in code cleanup

[<Joe...@t-...>]

Daniel,

Congratulations, too, for obtaining a GSOC!

> I'd like to change that. In order to do so I'd like to "modernize" the code
> base, starting with the C code. That means [...] splitting code into smaller units,

Speaking about lispbibl.d, I'm not sure it's too big, in absolute terms. I've learned
that the "best" file size very much depends on how we use a file, i.e. the tools
we use to manipulate it.

Back in history, approx. 25 years ago, when I first compiled CLISP on a 3MB(?) Amiga,
opening the huge - by old standards - 500kB lispbibl.d took at least once second,
just to load. Searching through 500kB was not instantaneous. Fast forward to the
present, searching in a 1MB file shows no lag.
However, editors are still an order of magnitude better at handling single files
than at working with multiple, related files. How would you specify a search in
only include files, excluding code files, e.g. record.d?

So having everything in once place still has benefits, today.

If you look closely, you'll still find comments like "ARRBIBL for ARRAY.D". Lispbibl.d
back then was a concatenation of those more specific include files. BIBL means
'Bibliothek', i.e. library.
We found out that, due to memory fragmentation issues, it was more reliable to
compile all C files using the one giant include file than the individual pieces.
That's no more an issue.

Last but not least, like it or not, trying to create a proper hierarchy of
tiny ordered includes costs a lot of effort and time. Unsurprisingly, all languages
have something to use forward references. I don't know if it's worth the effort
to change that.

Kind regards,
	Jörg Höhle

Re: GSoC 2016: proposal accepted!

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-04-23 08:54:31 +0000]:
> Sam writes:
>
>> I suggest that you clone the mercurial repo and commit your patches in
>> your branch.
>
> With "branch" you mean "branching by cloning"[1], right? (using mostly
> git, mercurial not having a "standard" branching approach was at first
> a bit shocking.)

whatever works for you.

> In order to have the code/patches accessible (and also safe) I'd like
> to put them to some remote repository: On sourceforge there's a "fork"
> button, which is probably doing exactly what is needed (creating a
> clone where I can push changes to). Or is there a better approach?

I don't know, but it sounds reasonable.

> A question regarding C coding style: src/CodingStyle has not much on this,
> when writing new code / fixing old code, what style should I try to follow?
> As far as I can see there's no "overall" consistent use of a single style.
> Should I try to follow the general GNU coding style[2]?

CLISP style is more geared towards "compactness" (e.g., only the closing
brace is ever on its own line, never the opening - unless it is the
first in a function and thus in the first column; no more than 2 spaces
for indentation &c).

> What's the "oldest" supported C standard? There's utils/varbrace.c to
> be able to write C99-like code but target C90, so I guess it's C90?

Yes, it has been like that for quite some time.

Your next step after making the release might be removing varbrace and
requiring C99 - are there any major compilers which don't support it?

> Should I keep the convention of using the var macro, or directly write
> C90 compatible code? As already said, I personally would choose the
> later, but it's no problem for me using var for declarations as
> currently done.

I suggest sticking with the current conventions for now.

> Then, regarding organization: Should we set up regular status reports?
> I guess I start building a (more) detailed list of what needs to be
> done, in order to be able to track progress.

If you have no questions for a week, please send e-mail here describing
your progress.
I am afraid you will be asking questions more frequently than sending
progress reports though :-)

> Being a GNU project, do you need any legal / licence related document
> from me?

Not really.  CLISP is GNU GPL v2+, so all your code is to be released
under that license; I assume you understand that. :-)
A simple note in an e-mail that you acknowledge and accept that fact and
do consent to releasing your code under GPLv2+ is fine.

> Is it ok if we keep this mailing list as "main communication channel"?

Yes, please!


-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11803000
http://www.childpsy.net/ http://www.dhimmitude.org http://camera.org
http://mideasttruth.com http://honestreporting.com http://truepeace.org
Time would have been the best Teacher, if it did not kill all its students.

Re: GSoC 2016: proposal accepted!

[Daniel J. <dan...@gm...>]

Sam writes:
> Your proposal
>
https://summerofcode.withgoogle.com/dashboard/project/6280060281552896/overview/
> has been formally accepted for Google Summer of Code 2016.
>
> Congratulations!

Thank you :) I'm really glad that it got accepted (especially considering
the
relatively low number of slots for GNU this year).

> I suggest that you clone the mercurial repo and commit your patches in
> your branch.

With "branch" you mean "branching by cloning"[1], right? (using mostly git,
mercurial not having a "standard" branching approach was at first a bit
shocking.)

In order to have the code/patches accessible (and also safe) I'd like to put
them to some remote repository: On sourceforge there's a "fork" button,
which is probably doing exactly what is needed (creating a clone where I can
push changes to). Or is there a better approach?

> This will require updating gnulib imports as necessary, merging in the
> various patches submitted over the years and fixing the windows
> socket/handle issue.

I'm thinking of starting directly off with the gnulib imports, as a lot of
code depends on them (and the last import seems to have broken stuff).
Makefile.devel pulls gnulib from git, I guess that's what I do, too? (There
are also "stable" releases, but AFAIK they aren't used that much.)

> Please ask questions here.

A question regarding C coding style: src/CodingStyle has not much on this,
when writing new code / fixing old code, what style should I try to follow?
As far as I can see there's no "overall" consistent use of a single style.
Should I try to follow the general GNU coding style[2]?

What's the "oldest" supported C standard? There's utils/varbrace.c to be
able
to write C99-like code but target C90, so I guess it's C90? Should I keep
the
convention of using the var macro, or directly write C90 compatible code? As
already said, I personally would choose the later, but it's no problem for
me
using var for declarations as currently done.

Then, regarding organization: Should we set up regular status reports? I
guess
I start building a (more) detailed list of what needs to be done, in order
to
be able to track progress.

Being a GNU project, do you need any legal / licence related document from
me?

Is it ok if we keep this mailing list as "main communication channel"?

> Good luck and thanks!

Thank you :)

[1] http://stevelosh.com/blog/2009/08/a-guide-to-branching-in-mercurial/
[2] https://www.gnu.org/prep/standards/html_node/Writing-C.html#Writing-C

GSoC 2016: proposal accepted!

[Sam S. <sd...@gn...>]

Daniel,

Your proposal
https://summerofcode.withgoogle.com/dashboard/project/6280060281552896/overview/
has been formally accepted for Google Summer of Code 2016.

Congratulations!

You now have 2 months until June 20, 2016 (Mid-Term Evaluation) to pass
the first hurdle: build CLISP on
1. linux (one flavor),
2. bsd (one flavor),
3. windows (mingw?).

This will require updating gnulib imports as necessary, merging in the
various patches submitted over the years and fixing the windows
socket/handle issue.

I suggest that you clone the mercurial repo and commit your patches in
your branch.
When you pass the first hurdle, you will submit the patches here, and we
will review and import them.

The rest of the summer will be dedicated to:
1. building for more linux and bsd flavors
2. building for macosx
3. building for arm & power architectures
4. making an actual release

Please ask questions here.

Good luck and thanks!

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11803000
http://www.childpsy.net/ http://jihadwatch.org http://www.dhimmitude.org
http://palestinefacts.org http://camera.org http://iris.org.il
Booze is the answer. I can't remember the question.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Sam Steingold writes:
> The approach of you refactoring many huge files and us reviewing your
> changes does not scale because none of us have time for that.

Oh, I see! That indeed wouldn't make much sense now.

>>> all files include lispbibl.d, so this means that all files use them. :-)
>>
>> Hehe, what I meant was more like: There seems to be versions of
>> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
>> that could be also provided by gnulib. That's what's confusing me: Are
>> these separate versions on purpose, or just not yet replaced by the
>> Gnulib ones?
>
> CLISP has a lot of code which is now available elsewhere because much of
> CLISP was written 20+ years ago.
>
> I see no problem in relying on external libraries.
> I am not sure that gnulib provides avl trees though.

Good. It does, there are three implementations, written by Bruno Haible, so
I'd guess they're probably very similar or even the same as the CLISP
implementation:
https://www.gnu.org/software/gnulib/MODULES.html#ansic_ext_container

>> What's the general course of action here? Fix it in gnulib (as far as
>> that's possible, eventually using it's local modules feature) or work
>> around "here"?
>
> I suggest that you discuss this on the gnulib mailing list.
> I have no idea what the gnulib people will recommend.

Ok, I'll do that (searching the list for preliminary information now).

> I added a couple minor comments, otherwise lgtm.

Thank you :)

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-03-25 04:59:01 +0100]:
>
> I hope this is now "close enough" to a proposal in order to be reviewed:
> https://docs.google.com/document/d/1VEwlAhA-YNEYiBHzz4jWS9BnRMdxgwF71ihqRdP58uY/edit?usp=sharing

I added a couple minor comments, otherwise lgtm.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://honestreporting.com http://iris.org.il
http://memri.org http://dhimmi.org http://camera.org
C combines the power of assembler with the portability of assembler.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-03-25 03:32:51 +0100]:
>
> Sam Steingold writes:
>> this approach does not scale.
>
> Can you elaborate on that a bit?

The approach of you refactoring many huge files and us reviewing your
changes does not scale because none of us have time for that.

>> all files include lispbibl.d, so this means that all files use them. :-)
>
> Hehe, what I meant was more like: There seems to be versions of
> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
> that could be also provided by gnulib. That's what's confusing me: Are
> these separate versions on purpose, or just not yet replaced by the
> Gnulib ones?

CLISP has a lot of code which is now available elsewhere because much of
CLISP was written 20+ years ago.

I see no problem in relying on external libraries.
I am not sure that gnulib provides avl trees though.

> What's the general course of action here? Fix it in gnulib (as far as
> that's possible, eventually using it's local modules feature) or work
> around "here"?

I suggest that you discuss this on the gnulib mailing list.
I have no idea what the gnulib people will recommend.

> What priority do non cygwin builds on windows have? Targeting mingw or
> the microsoft compilers?

Up to you - whatever compiler is easier to use/gives better speed/size
optimization &c.
I guess mingw is easier because it's the same gcc.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://truepeace.org http://jihadwatch.org
http://dhimmi.org http://think-israel.org http://honestreporting.com
War has never solved anything - except for ending Slavery, Nazism, Communism.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

I hope this is now "close enough" to a proposal in order to be reviewed:
https://docs.google.com/document/d/1VEwlAhA-YNEYiBHzz4jWS9BnRMdxgwF71ihqRdP58uY/edit?usp=sharing

Daniel

2016-03-25 3:32 GMT+01:00 Daniel Jour <dan...@gm...>:
> Hi Sam,
>
> Sam Steingold writes:
>> this approach does not scale.
>
> Can you elaborate on that a bit? As I already wrote to Don, it's just
> the way I'm used to, based on previous experiences, but I'm completely
> open to learn :) (I'm completely self-taught and have never received
> any mentoring or - real, useful - teaching in "programming").
>
>
>> all files include lispbibl.d, so this means that all files use them. :-)
>
> Hehe, what I meant was more like: There seems to be versions of
> alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
> that could be also provided by gnulib. That's what's confusing me: Are
> these separate versions on purpose, or just not yet replaced by the
> Gnulib ones?
>
>>>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>>>    Windows uses Handles)
>>>
>>> Can you provide a bit more background on that? Is this related to
>>> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
>>> thus a general gnulib issue? Or does this only affect CLISP?
>>
>> this is a general property of gnulib.
>> clisp used to work around this, but the latest import of gnulib files (5
>> years ago?) broke it.
>> IIRC, clisp no longer builds on windows.
>
> I think it builds on cygwin with Ken's patches, but not "native" windows.
> I'll try to get a windows machine / vm up in order to check this.
>
> What's the general course of action here? Fix it in gnulib (as far as that's
> possible, eventually using it's local modules feature) or work around "here"?
>
> What priority do non cygwin builds on windows have? Targeting mingw or
> the microsoft compilers?
>
>
>>> How "extensive" are these self-tests?
>>
>> there are a lot of them and they cover most of the code.
>> (modules have their own tests).
>
> I found them, they're awesome.
>
>
>>> I guess I should get a proposal ready ASAP then.
>>
>> please do!
>
> Work is in progress, I have rough version that's ready for sharing soon.
> Due to the limited time left I add some notes here to start with:
>
> -- Title: CLISP - Make a release (finally)
> -- Summary:
> CLISP has seen its last release in 2010, almost 6 years ago.
> Since then there have been a lot of changes, partly implemented features
> and bug fixes. In combination with an import of Gnulib modules this
> resulted in a code base that is no longer portable and not suitable for a
> release. The purpose of this project is to fix the portability issues and to
> cleanly include suitable Gnulib modules, allowing a new release with the
> improvements of the past years to be published.
>
> --Raw notes regarding plan:
> * Get a complete picture of the code, especially what's broken on non linux
> platforms.
> * Get rid of (probably? almost) everything that can be provided by a suitable
> gnulib module, using the most up to date version of gnulib (replace one part,
> then run self tests, update doc, repeat). Potentially extending self tests.
> * Fix gnulib on Windows. At least "enough" for CLISP to work.
> * Mid-term: core CLISP builds and passes self tests on Linux, BSD, Mac and
> Windows.
> * update CLISP modules
> * include useful (not yet used) patches and changes. It might be worthwhile to
> think about holding back (possibly reverting) some half-finished changes here,
> keeping them for a next release until they're finished (multithreading? jit?)
> * general clean up of code and implementation notes ("polishing for the
> release", so to say)
> * Finally: release.
>
> -- Skills I already “have”: C and Lisp, knowledge of memory management and
> garbage collection as well as general VM/bytecode design and compiler
> building. GNU make, Shell programming, general Linux competence.
>
> -- Things I need to learn: Proper usage (as a developer) of the autotools and
> the GNU build system, “hands on experience” regarding release process(es),
> probably understanding CLISP's VM in depth will help, and finally how to
> achieve portability to a lot of platforms. Update of my knowledge of windows
> programming (it's been a while).
>
> Sam, do you think that I should include references to previous work? Most
> helpful is probably my stackoverflow.com account .. since my github repositories
> ... ehm .. aren't exactly neat and tidy and contain only a small
> (largely outdated)
> fraction of the stuff I've been doing all the years :)
>
> Regarding communication: I'm best reached via email, but starting early-mid
> April irc becomes an option, too, though I'm constrained regarding time of day.
> This might be an issue if you'd like (very) frequent real time communication.
> There's not much planned during the coding (and bonding) time span, apart
> from a possible holiday of probably a week early August. My exams are at the
> end of July, though that won't interfere (I'm thankfully learning fast
> and was able
> to complete this semesters exams with top scores without being present
> or investing
> much time during the semester. This has been the same for the last semesters, so
> I'm very sure that this will work.)
>
> Do I need to explicitly mention you in the proposal, or is this GNU
> internally sorted out?
>
> Looking forward to some feedback (if still possible, I know that only a bit more
> than 16 hours left isn't exactly "much time")
>
> Daniel
>
>> --
>> Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
>> http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
>> http://honestreporting.com http://truepeace.org http://think-israel.org
>> God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Hi Sam,

Sam Steingold writes:
> this approach does not scale.

Can you elaborate on that a bit? As I already wrote to Don, it's just
the way I'm used to, based on previous experiences, but I'm completely
open to learn :) (I'm completely self-taught and have never received
any mentoring or - real, useful - teaching in "programming").


> all files include lispbibl.d, so this means that all files use them. :-)

Hehe, what I meant was more like: There seems to be versions of
alloca, an avl tree, sorting code, safe malloc and more in CLISP's code
that could be also provided by gnulib. That's what's confusing me: Are
these separate versions on purpose, or just not yet replaced by the
Gnulib ones?

>>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>>    Windows uses Handles)
>>
>> Can you provide a bit more background on that? Is this related to
>> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
>> thus a general gnulib issue? Or does this only affect CLISP?
>
> this is a general property of gnulib.
> clisp used to work around this, but the latest import of gnulib files (5
> years ago?) broke it.
> IIRC, clisp no longer builds on windows.

I think it builds on cygwin with Ken's patches, but not "native" windows.
I'll try to get a windows machine / vm up in order to check this.

What's the general course of action here? Fix it in gnulib (as far as that's
possible, eventually using it's local modules feature) or work around "here"?

What priority do non cygwin builds on windows have? Targeting mingw or
the microsoft compilers?


>> How "extensive" are these self-tests?
>
> there are a lot of them and they cover most of the code.
> (modules have their own tests).

I found them, they're awesome.


>> I guess I should get a proposal ready ASAP then.
>
> please do!

Work is in progress, I have rough version that's ready for sharing soon.
Due to the limited time left I add some notes here to start with:

-- Title: CLISP - Make a release (finally)
-- Summary:
CLISP has seen its last release in 2010, almost 6 years ago.
Since then there have been a lot of changes, partly implemented features
and bug fixes. In combination with an import of Gnulib modules this
resulted in a code base that is no longer portable and not suitable for a
release. The purpose of this project is to fix the portability issues and to
cleanly include suitable Gnulib modules, allowing a new release with the
improvements of the past years to be published.

--Raw notes regarding plan:
* Get a complete picture of the code, especially what's broken on non linux
platforms.
* Get rid of (probably? almost) everything that can be provided by a suitable
gnulib module, using the most up to date version of gnulib (replace one part,
then run self tests, update doc, repeat). Potentially extending self tests.
* Fix gnulib on Windows. At least "enough" for CLISP to work.
* Mid-term: core CLISP builds and passes self tests on Linux, BSD, Mac and
Windows.
* update CLISP modules
* include useful (not yet used) patches and changes. It might be worthwhile to
think about holding back (possibly reverting) some half-finished changes here,
keeping them for a next release until they're finished (multithreading? jit?)
* general clean up of code and implementation notes ("polishing for the
release", so to say)
* Finally: release.

-- Skills I already “have”: C and Lisp, knowledge of memory management and
garbage collection as well as general VM/bytecode design and compiler
building. GNU make, Shell programming, general Linux competence.

-- Things I need to learn: Proper usage (as a developer) of the autotools and
the GNU build system, “hands on experience” regarding release process(es),
probably understanding CLISP's VM in depth will help, and finally how to
achieve portability to a lot of platforms. Update of my knowledge of windows
programming (it's been a while).

Sam, do you think that I should include references to previous work? Most
helpful is probably my stackoverflow.com account .. since my github repositories
... ehm .. aren't exactly neat and tidy and contain only a small
(largely outdated)
fraction of the stuff I've been doing all the years :)

Regarding communication: I'm best reached via email, but starting early-mid
April irc becomes an option, too, though I'm constrained regarding time of day.
This might be an issue if you'd like (very) frequent real time communication.
There's not much planned during the coding (and bonding) time span, apart
from a possible holiday of probably a week early August. My exams are at the
end of July, though that won't interfere (I'm thankfully learning fast
and was able
to complete this semesters exams with top scores without being present
or investing
much time during the semester. This has been the same for the last semesters, so
I'm very sure that this will work.)

Do I need to explicitly mention you in the proposal, or is this GNU
internally sorted out?

Looking forward to some feedback (if still possible, I know that only a bit more
than 16 hours left isn't exactly "much time")

Daniel

> --
> Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
> http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
> http://honestreporting.com http://truepeace.org http://think-israel.org
> God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Daniel J. <dan...@gm...>]

Hi Don,

Don Cohen writes:
>
> Daniel Jour writes:
>  > > > Looking at the code base, I think this low activity stems
> My interpretation has been that the low activity is largely due to the
> code being GOOD - although I've found things I'd like to change, these
> have been easy enough to work around to make the cost/benefit favor
> leaving them.

Please don't get me wrong: The code is good. It's implementing good logic,
making mostly sensible use of the available tools of the C language (I'm
focusing on the runtime implementation). But:

 * The files are huge and contain way too much. Related stuff, but in
principle stuff that makes up separate parts, and could and should be
kept separate.

 * Many functions are also huge, and contain really a lot sections that
are subject to conditional compilation, depending on the target platform
and the selected configuration.

The implication is that in order to make well informed changes to one
part of the code, you also need to know how it interacts with the rest
of the code. If there would be a clean interface then this is not a problem,
because the interface should clearly describe these interactions.

Assume a hobbyist user found a bug in the parser. She's presented with
a 10k+ lines src/io.d which includes (the result of preprocessing) a 17k+
lines src/lispbibl.d. That's a lot of code to even quickly skim over, in order
to make sure not to introduce any unwanted side effects. The
implementation notes help a lot, but provide more of a high level overview,
not so much the interactions between the various code parts.

I think this difficulty to get working with the code is demonstrated by the
low development activity, the age and number of open bugs and the low
number of feature requests (general tail call optimization?).

 * There seems to be a lot of code determining the "available features"
and the "configuration" of the target machine which should be, to at
least some extend, handled by the C library respectively gnulib.

 * Undefined behavior, probably mostly in spvw* and also sort.d:
Stuff like comparing (less, greater) pointers, out of bounds indexing
and some cast just aren't legal C, and can lead to all sorts of nasty bugs,
especially when using an optimizing compiler. I'd guess that's the reason
that (e.g.) my distributions build instructions pass in CFLAGS=-O0.

There's probably more in the direction of general code readability. (A
fun thing to do is the following: Collect the names of all local variables
per function of a source file and the names of the functions. Then shuffle
and try to figure out which set of local variables is part of which function.)

I hope the above didn't make a harsh or arrogant impression, it's basically
how I start when e.g. reviewing code on stackoverflow, and based on the
experiences I made in the last 14 years. I don't assume it to be the "only
way", and I'm completely open to revising my view :) After all, I've never
received any mentoring or formal teaching in programming (I'm not
counting my university ... I can tell with certainty now that universities are
not places that teach "how to code").

> Over the years I've suggested a few small changes on the mailing
> list that never made it as far as patches, partly cause I didn't even
> know that patches were being collected.
> I'd appreciate if some of those could be included when the patches are
> incorporated.

I will search for them in the mailing lists. The more input, the better, IMO.
SF doesn't support searching for names, though, and the search provided
by gmane seems to be working, but "open ended", so it's probable that
I miss some posts. Do you have any "record" of your suggestions?


> Just wondering, how much interest is there in different linux versions,
> and especially old ones?  Some of my nightly test builds are on
> Fedora Core release 4 (2.6.17-1.2142_FC4).

AFAIK the "core" of CLISP doesn't have much dependencies. libsigsegv,
libffi and libffcall (and perhaps readline) are listed as dependencies in
my package description, ldd only shows glibc and pthreads. Assuming
proper use of gnulib CLISP should probably be able to run on most of the
platforms supported by gnulib, which would also include older linux
versions.

CLISP is also known for its portability (and availability, IIRC it was part of
the default installation of debian in the past. At least it's on every machine
in my universities computer pools, and AFAIK the system administrator
hasn't installed it on purpose) so IMO it would be good if a new release
wouldn't break this backwards compability.

But, being completely new here, that's just me guessing :)


>  > How "extensive" are these self-tests?
> You'll see.
> I consider them to be very extensive.
> Sometimes they begin to feel interminable.

Yes, I had the time to read over some of them now. That's basically a huge
suite of regression tests, and that's awesome :)

Daniel

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Daniel Jour <qnavry.bregjvt@tznvy.pbz> [2016-03-23 08:41:00 +0000]:
>
> That's great. I agree, modernizing the complete code base is a task that's
> too huge to attempt as a first CLISP project. What I hoped for was that it's
> ok if, picking a random example, readtable related code got extracted from
> src/io.d, i.e. it's ok when runtime code is refactored to some degree.

this approach does not scale.

>> 1. update all gnulib imports
>
> To what extend is CLISP using gnulib atm? I've found the
> gnulib-imported target in Makefile.devel (as well as related stuff),
> but am unsure how far the actual src/* files make use of it. As far as
> I can tell only src/lispbibl.d, src/_clisp.c and src/glmalloc.c seem
> to use it (only taking the runtime into approach, not any modules).

all files include lispbibl.d, so this means that all files use them. :-)

>> 2. fix gnulib/Windows interaction (gnulib assumes FD to be an int,
>>    Windows uses Handles)
>
> Can you provide a bit more background on that? Is this related to
> http://lists.gnu.org/archive/html/bug-gnulib/2016-03/msg00033.html and
> thus a general gnulib issue? Or does this only affect CLISP?

this is a general property of gnulib.
clisp used to work around this, but the latest import of gnulib files (5
years ago?) broke it.
IIRC, clisp no longer builds on windows.

> How "extensive" are these self-tests?

there are a lot of them and they cover most of the code.
(modules have their own tests).

> I guess I should get a proposal ready ASAP then.

please do!


-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://memri.org http://islamexposedonline.com
http://honestreporting.com http://truepeace.org http://think-israel.org
God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[Sam S. <sd...@gn...>]

> * Don Cohen <qba...@vf...3-vap.pbz> [2016-03-23 11:38:39 +0000]:
>
> Also how much interest is there in different build configurations?
> One of the configurations on that old machine,
>  ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock build-mt
> ends up with
>  ./clisp-link: line 97:   538 Segmentation fault      "$@"
> (and has for many years now)
> This one builds, though:
>  ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock --with-dynamic-modules=no build-mt-no

both dynamic modules and threads are important features.
(threads are still experimental though.)
the segmentation fault above is certainly a bug, and should be fixed.
http://www.cygwin.com/acronyms/#PTC

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 15.10 (wily) X 11.0.11702000
http://www.childpsy.net/ http://mideasttruth.com http://honestreporting.com
http://jihadwatch.org http://islamexposedonline.com http://camera.org
God had a deadline, so He wrote it all in Lisp.

Re: clisp - general interest in code cleanup

[<don...@is...>]

Daniel Jour writes:
 > > > Looking at the code base, I think this low activity stems 
My interpretation has been that the low activity is largely due to the
code being GOOD - although I've found things I'd like to change, these
have been easy enough to work around to make the cost/benefit favor
leaving them.

 > I didn't know about the patches section of sf.net until Ken Brown brought
 > it up (https://sf.net/p/clisp/mailman/message/34957653/).

It's news to me too.  In fact I only recently found out that the hg
I had been using for years was not the current one.
Using an old hg makes the activity seem even lower!
I'm still doing several nightly builds to check that any changes don't
break either clisp builds or builds of my own software on top of clisp.

Over the years I've suggested a few small changes on the mailing
list that never made it as far as patches, partly cause I didn't even
know that patches were being collected.  
I'd appreciate if some of those could be included when the patches are
incorporated.

 > > 3. make sure that CLISP builds and passes self-tests on as many
 > >    platforms as possible, but at least
 > >    A. Linux (fedora, ubuntu &c)
 > >    B. Windows (whatever the most common version is now, with and without
 > >       cygwin)
 > >    C. Mac OS X
 > >    D. *BSD (FreeBSD, OpenBSD, NetBSD)

Just wondering, how much interest is there in different linux versions,
and especially old ones?  Some of my nightly test builds are on
Fedora Core release 4 (2.6.17-1.2142_FC4).

Also how much interest is there in different build configurations?
One of the configurations on that old machine,
 ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock build-mt
ends up with
 ./clisp-link: line 97:   538 Segmentation fault      "$@"
(and has for many years now)
This one builds, though:
 ./configure --with-debug --with-threads=POSIX_THREADS --with-module=rawsock --with-dynamic-modules=no build-mt-no

 > How "extensive" are these self-tests?
You'll see.
I consider them to be very extensive.
Sometimes they begin to feel interminable.