clisp-devel

status update

[Sam S. <sd...@po...>]

Daniel,
the mid-term eval is now.
what is your status?
did you manage to build clisp with rawsock on windows?
if yes, please build a binary distribution ("make -f Makefile.devel distrib")
and make the binary available for download.
everyone with access to a windows box: please download and test.
alas, I am on vacation with sporadic internet access at best, so I
would appreciate any help I could get.
thanks

-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net/>

Re: status update

[Daniel J. <dan...@gm...>]

> what is your status?

I was struggling with the build system, but I think I got it fixed
now. Changes to the actual code seem (so far) to work flawlessly.  I
had a bit less time available the last week, which is why I'm a bit
behind schedule.

> did you manage to build clisp with rawsock on windows?

Not yet (completely), but I think I'll be able to in about 24 hours.

> if yes, please build a binary distribution ("make -f Makefile.devel
> distrib") and make the binary available for download.

I do that as soon as it builds and passes the tests, which will be
hopefully very soon.

Re: status update

[Sam S. <sd...@po...>]

Daniel,

>> The deadline to complete your midterm evaluation is Monday, June 27th at 19:00 UTC

By that time I need to have an independent confirmation that you built
clisp on windows with working rawsock module (that was the condition
for the midterm "pass" from the get go).
Please make sure that I do get the confirmation by then.
Sorry if that sounds harsh, but rules are rules.

On Sat, Jun 25, 2016 at 10:19 AM, Daniel Jour <dan...@gm...> wrote:
>> what is your status?
>
> I was struggling with the build system, but I think I got it fixed
> now. Changes to the actual code seem (so far) to work flawlessly.  I
> had a bit less time available the last week, which is why I'm a bit
> behind schedule.

what are the changes you had to make to get it working?
it's important that the changes "scale", i.e., a gnulib update does
not entail much work.

>> did you manage to build clisp with rawsock on windows?
>
> Not yet (completely), but I think I'll be able to in about 24 hours.
>
>> if yes, please build a binary distribution ("make -f Makefile.devel
>> distrib") and make the binary available for download.
>
> I do that as soon as it builds and passes the tests, which will be
> hopefully very soon.

Thanks!

-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net/>

Re: status update

[Daniel J. <dan...@gm...>]

Oh my, I've been working on fixing something that's just not fixable
for the last day (reverse DNS lookup ... doesn't work on my test
windows machine, not even with nslookup.exe).

Source tarbal:
https://drive.google.com/file/d/0B303fG5hcio_bzR4SWszcnRoTWM/view?usp=sharing


> By that time I need to have an independent confirmation that you
> built clisp on windows with working rawsock module (that was the
> condition for the midterm "pass" from the get go).  Please make sure
> that I do get the confirmation by then.  Sorry if that sounds harsh,
> but rules are rules.

No problem. I'm a week behind schedule, and there's absolutely nothing
that you could've done / helped me with to prevent that.

CLISP with rawsock builds on windows, but it's not yet able to pass
all tests. The core tests hangs when trying to let another CLISP
instance send data via sockets to itself, but as far as I can see this
is an issue with the invocation of that second instance.

I removed the reverse DNS lookup from the rawsock tests, that way I
got the number of errors down to 13. These seem mostly to come from
the code that is sitting in syscalls module and core CLISP. On another
windows test machine rawsock tests give a stack overflow when testing
rawsock errors (the test that starts with "(block foo"), I don't know
why yet.

Regarding the independent confirmation: I'm a bit afraid that due to
the lack of remaining time this could be an issue (I don't know how
many are actively following this conversation). Is another GSoC
student independent enough? (I was thinking about asking for help on
the summer of code mailing list .. in order to have someone download
the source tarbal, build it and send the confirmation to clisp-list.)


> what are the changes you had to make to get it working?  it's
> important that the changes "scale", i.e., a gnulib update does not
> entail much work.

The changes to rawsock? Not that much, actually. I switched its build
system to one built with autotools, mostly in order to give it an
gnulib checkout of its own. I'm writing some gnulib modules to patch
functionality missing on windows (ifaddrs.h functions atm) so that the
code in rawsock can be completelty platform independent.

If by "scale" you mean that they can be built upon: Updating gnulib (when,
for example there's an update in a used module) means running

gnulib-tool --update

in modules/rawsock, followed by

autoreconf --force --install

This is the exact procedure I'm following whenever I've made a change
to one of the used modules.

Though, atm the code is messy: I copied some parts from core CLISP and
the syscalls module, although I don't use it. The reason is that both
core CLISP as well as syscalls are still using the old, not yet
updated gnulib code. Updating that failed due to the build system not
being able to "rebuild" itself (the configures target in
Makefile.devel fails).

Looking at how much time I wasted struggling with the build system I'd
like to replace it with an "standard" autotools one. I already started
work on that, it shouldn't be much of an issue.

I'd like to start a discussion about the modules and their
organization in general, but I'll write that in a separate message
soon.

Re: status update

[Sam S. <sd...@po...>]

Hi Daniel,

Congratulations, you pass the midterm evaluation.

Now, please note that you really are behind the schedule.
You need to polish your fixes to rawsock, get your new modules
accepted to gnulib, finish converting clisp to use autotools
throughout, and make a release.
(note that your tar ball is missing version number :-)

Please keep sending updates a couple of times a week.

Good luck and thanks!

On Mon, Jun 27, 2016 at 6:06 AM, Daniel Jour <dan...@gm...> wrote:
> Oh my, I've been working on fixing something that's just not fixable
> for the last day (reverse DNS lookup ... doesn't work on my test
> windows machine, not even with nslookup.exe).
>
> Source tarbal:
> https://drive.google.com/file/d/0B303fG5hcio_bzR4SWszcnRoTWM/view?usp=sharing
>
>
>> By that time I need to have an independent confirmation that you
>> built clisp on windows with working rawsock module (that was the
>> condition for the midterm "pass" from the get go).  Please make sure
>> that I do get the confirmation by then.  Sorry if that sounds harsh,
>> but rules are rules.
>
> No problem. I'm a week behind schedule, and there's absolutely nothing
> that you could've done / helped me with to prevent that.
>
> CLISP with rawsock builds on windows, but it's not yet able to pass
> all tests. The core tests hangs when trying to let another CLISP
> instance send data via sockets to itself, but as far as I can see this
> is an issue with the invocation of that second instance.
>
> I removed the reverse DNS lookup from the rawsock tests, that way I
> got the number of errors down to 13. These seem mostly to come from
> the code that is sitting in syscalls module and core CLISP. On another
> windows test machine rawsock tests give a stack overflow when testing
> rawsock errors (the test that starts with "(block foo"), I don't know
> why yet.
>
> Regarding the independent confirmation: I'm a bit afraid that due to
> the lack of remaining time this could be an issue (I don't know how
> many are actively following this conversation). Is another GSoC
> student independent enough? (I was thinking about asking for help on
> the summer of code mailing list .. in order to have someone download
> the source tarbal, build it and send the confirmation to clisp-list.)
>
>
>> what are the changes you had to make to get it working?  it's
>> important that the changes "scale", i.e., a gnulib update does not
>> entail much work.
>
> The changes to rawsock? Not that much, actually. I switched its build
> system to one built with autotools, mostly in order to give it an
> gnulib checkout of its own. I'm writing some gnulib modules to patch
> functionality missing on windows (ifaddrs.h functions atm) so that the
> code in rawsock can be completelty platform independent.
>
> If by "scale" you mean that they can be built upon: Updating gnulib (when,
> for example there's an update in a used module) means running
>
> gnulib-tool --update
>
> in modules/rawsock, followed by
>
> autoreconf --force --install
>
> This is the exact procedure I'm following whenever I've made a change
> to one of the used modules.
>
> Though, atm the code is messy: I copied some parts from core CLISP and
> the syscalls module, although I don't use it. The reason is that both
> core CLISP as well as syscalls are still using the old, not yet
> updated gnulib code. Updating that failed due to the build system not
> being able to "rebuild" itself (the configures target in
> Makefile.devel fails).
>
> Looking at how much time I wasted struggling with the build system I'd
> like to replace it with an "standard" autotools one. I already started
> work on that, it shouldn't be much of an issue.
>
> I'd like to start a discussion about the modules and their
> organization in general, but I'll write that in a separate message
> soon.



-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net/>

Re: status update

[Pascal J. B. <pj...@in...>]

Daniel Jour <dan...@gm...> writes:

> Regarding the independent confirmation: I'm a bit afraid that due to
> the lack of remaining time this could be an issue (I don't know how
> many are actively following this conversation). Is another GSoC
> student independent enough? (I was thinking about asking for help on
> the summer of code mailing list .. in order to have someone download
> the source tarbal, build it and send the confirmation to clisp-list.)

I can boot a MS-Windows-7 computer here, so when you have a binary ready
post its URL and I'll try it out on Friday night or Sunday.

-- 
__Pascal Bourguignon__                 http://www.informatimago.com/
“The factory of the future will have only two employees, a man and a
dog. The man will be there to feed the dog. The dog will be there to
keep the man from touching the equipment.” -- Carl Bass CEO Autodesk

status update

[<Joe...@t-...>]

Hi,

I can test on MS-Windows 7 too (and would be happy to try out a recent clisp), but not before the 11th of July...

Keep up the good work,
	Jörg Höhle

Re: status update

[Daniel J. <dan...@gm...>]

Sam wrote:
> Congratulations, you pass the midterm evaluation.

Thank you :)


> Now, please note that you really are behind the schedule.  You need
> to polish your fixes to rawsock, get your new modules accepted to
> gnulib, finish converting clisp to use autotools throughout, and
> make a release.

I'm currently working on getting the new modules ready for submission
to gnulib. The m4/autoconf code is the hardest part here (I haven't
written that much of m4 before).

More specifically, I:

* wrote an ifaddrs.h module for gnulib

* wrote an getifaddrs module (implementing functions from above
 header) for gnulib, initially using WSAIoctl, just to learn that this
 happily returns an empty list of interfaces without any indication of
 failure most of the time, so I rewrote it to use GetAdaptersAddresses
 instead.

* am writing a gnulib module to implement if_nameindex (and related
  functions) on mingw/Windows.

It also took me some time to understand why AC_SEARCH_LIBS isn't
working with Windows dlls/libs: The "stdcall" calling convention
decorates (function) symbols with the number of bytes used by the
arguments on the stack; but the autoconf macro generates an
declaration (with cdecl calling convention) like

char some_function();

so the test always fails with a linker error.


> (note that your tar ball is missing version number :-)

You mean because it still reads "2.49+" and not yet 2.5?


> Please keep sending updates a couple of times a week.

Please excuse that it's already a week since my last mail. I tried to
only send an update when some subtaks is finished, but I'll switch to
keeping you better informed about what I'm doing (even if it's not yet
finished).

Re: status update

[Sam S. <sd...@gn...>]

Hi Daniel,

> * Daniel Jour <qna...@tz...> [2016-07-04 14:08:17 +0200]:
> Sam wrote:
>
>> Please keep sending updates a couple of times a week.
>
> Please excuse that it's already a week since my last mail. I tried to
> only send an update when some subtaks is finished, but I'll switch to
> keeping you better informed about what I'm doing (even if it's not yet
> finished).

It has been almost a month(!) without any updates.
Are you still working on CLISP?
Thanks

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://www.memritv.org http://openvotingconsortium.org
http://think-israel.org http://www.dhimmitude.org http://iris.org.il
My name is Deja Vu. Have we met before?

Re: status update

[Daniel J. <dan...@gm...>]

Hi Sam,

> It has been almost a month(!) without any updates.

I'm very sorry for not contacting you earlier: I'm having health
issues since about 3 weeks (not minor, but hopefully and probably not
major issues). This significantly limited the amount of work I was
able to do, and - although it gets better - still forces me to take
little steps.


> Are you still working on CLISP?

Yes. Though due to the above I'm only slowly making progress :( I'm
still working on switching the configuration to an autotools based
system, as well as porting network related code to Windows. I've also
updated gnulib code (for all of CLISP): This is not yet building, but
already showed me which parts of CLISP will need to be adjusted. (I
know where the incompabilities are and how to fix most of them, but
did not yet fix them.)


Since I now receive (better than before) medical treatment I really
hope for a fast recovery. I'll know more tomorrow.


Best,
Daniel

Re: status update

[Sam S. <sd...@gn...>]

Thanks Daniel, get well soon!

On Wed, Aug 3, 2016 at 7:03 PM, Daniel Jour <dan...@gm...> wrote:
> Hi Sam,
>
>> It has been almost a month(!) without any updates.
>
> I'm very sorry for not contacting you earlier: I'm having health
> issues since about 3 weeks (not minor, but hopefully and probably not
> major issues). This significantly limited the amount of work I was
> able to do, and - although it gets better - still forces me to take
> little steps.
>
>
>> Are you still working on CLISP?
>
> Yes. Though due to the above I'm only slowly making progress :( I'm
> still working on switching the configuration to an autotools based
> system, as well as porting network related code to Windows. I've also
> updated gnulib code (for all of CLISP): This is not yet building, but
> already showed me which parts of CLISP will need to be adjusted. (I
> know where the incompabilities are and how to fix most of them, but
> did not yet fix them.)
>
>
> Since I now receive (better than before) medical treatment I really
> hope for a fast recovery. I'll know more tomorrow.
>
>
> Best,
> Daniel



-- 
Sam Steingold <http://sds.podval.org> <http://www.childpsy.net/>

Re: status update

[Daniel J. <dan...@gm...>]

Hi,

> Thanks Daniel, get well soon!

Thank you, it's getting better, I hope to be back to full health soon.

Quick questions:

*) Is WIDE_AUXI still in use / of interest? It was introduced
 2002-12-04 by Bruno, but as far as I can tell won't get used unless
 explicitly requested during configure via setting -D WIDE_AUXI in
 CFLAGS.

*) Bug report numbers. At various places in the code there are bug
 report numbers (like 710737, lispbibl.d, "Global register
 declarations"). I guess these numbers are from an abandoned bug
 tracker? Is there a backup of these bug reports somewhere? (I'm
 mostly just curious, there's no real need for these reports atm.)

*) intparam.h, floatparam.h: Instead of creating these headers during
 the build, I just wrote them using standard (C99) features (for
 example FLT_MANT_DIG from float.h). This makes cross-building much
 easier (did it even work before?). One thing that surprised me was
 that intparam.h contains endianness defines for each of short, int,
 long and long long. Since all these defines get "merged" into a
 single BIG_ENDIAN_P define, I guess systems with different endianness
 for different sized integer types are no longer supported? (Did we actually
 support such systems, like the PDP-11, at some point?)

*) Not GSoC related: If I understood this correctly, the math code
 backing CLISPs number system has at some point been extracted as a
 library, CLN (http://www.ginac.de/CLN/). Would it make sense to
 "throw out" our math code and to use that library instead? Looking at
 its code it has probably seen massive rewrites: It's now written in
 C++. It seems to be actively maintained and suitable for an ANSI CL
 number system.

Best,
Daniel

Re: status update

[Sam S. <sd...@gn...>]

Hi,

> * Daniel Jour <qna...@tz...> [2016-08-08 02:30:01 +0200]:
>
> *) Is WIDE_AUXI still in use / of interest? It was introduced
>  2002-12-04 by Bruno, but as far as I can tell won't get used unless
>  explicitly requested during configure via setting -D WIDE_AUXI in
>  CFLAGS.

I have no idea why Bruno introduced it (we already have WIDE_SOFT).
Unless he explains, I think we can discard it.

> *) Bug report numbers. At various places in the code there are bug
>  report numbers (like 710737, lispbibl.d, "Global register
>  declarations"). I guess these numbers are from an abandoned bug
>  tracker? Is there a backup of these bug reports somewhere? (I'm
>  mostly just curious, there's no real need for these reports atm.)

These numbers refer to the old SF bug tracker.
See emacs/misc.el:clisp-bug-reference-url-format for a hint on how to
convert those numbers to the new ones.

> *) intparam.h, floatparam.h: Instead of creating these headers during
> the build, I just wrote them using standard (C99) features (for
> example FLT_MANT_DIG from float.h).

Whatever you find easy to maintain.

> This makes cross-building much easier (did it even work before?).

I am sure it did :-)

> One thing that surprised me was that intparam.h contains endianness
> defines for each of short, int, long and long long. Since all these
> defines get "merged" into a single BIG_ENDIAN_P define, I guess
> systems with different endianness for different sized integer types
> are no longer supported? (Did we actually support such systems, like
> the PDP-11, at some point?)

You would be surprised by the zoo of systems once supported by CLISP.

> *) Not GSoC related: If I understood this correctly, the math code
>  backing CLISPs number system has at some point been extracted as a
>  library, CLN (http://www.ginac.de/CLN/). Would it make sense to
>  "throw out" our math code and to use that library instead? Looking at
>  its code it has probably seen massive rewrites: It's now written in
>  C++. It seems to be actively maintained and suitable for an ANSI CL
>  number system.

It might.

Note that all these issues should be postponed until AFTER you make the
release.
The ONLY focus of the release is making it work on all 3 major platforms
OOTB.



-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://camera.org http://memri.org
http://openvotingconsortium.org http://thereligionofpeace.com
Sufficiently advanced stupidity is indistinguishable from malice.

Re: WIDE_AUXI

[Bruno H. <br...@cl...>]

Daniel, Sam,

> > *) Is WIDE_AUXI still in use / of interest? It was introduced
> >  2002-12-04 by Bruno, but as far as I can tell won't get used unless
> >  explicitly requested during configure via setting -D WIDE_AUXI in
> >  CFLAGS.
> 
> I have no idea why Bruno introduced it (we already have WIDE_SOFT).

I think it had two purposes:
  - It was an experiment, to help developing the WIDE_SOFT memory model.
    For example, if a gcc bug or GC bug was hampering WIDE_SOFT, to
    see whether it occurred also with the (less demanding) WIDE_AUXI.
  - It has immediate (= not heap-allocated) single-floats, so it could
    be useful for numerical programs that use single-floats on 32-bit
    platforms.
    But such numerical programs more often use double-floats, and
    nowadays 64-bit platforms are the norm.

> Unless he explains, I think we can discard it.

Yes you can discard it.

Bruno

Re: WIDE_AUXI

[Sam S. <sd...@gn...>]

> * Bruno Haible <oe...@py...> [2016-08-09 10:13:11 +0200]:
>
> Yes you can discard it.

done

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://palestinefacts.org http://www.dhimmitude.org
http://www.memritv.org http://memri.org http://dhimmi.org
Yeah, yeah, I love cats too... wanna trade recipes?

Re: status update

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-07-04 14:08:17 +0200]:
>
> Sam wrote:
>> Congratulations, you pass the midterm evaluation.
>
> Thank you :)

You are welcome.

Next Monday, August 15, the final evaluation starts.

Are you prepared to make a release?

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.04 (xenial) X 11.0.11803000
http://www.childpsy.net/ http://americancensorship.org
http://think-israel.org http://truepeace.org http://memri.org
Only adults have difficulty with child-proof caps.

Re: status update

[Daniel J. <dan...@gm...>]

> Are you prepared to make a release?

Almost, but not yet. Unfortunately.

I really hope to get my work releaseable within the next (few)
days. I'm making good progress, but the delay from the midterm and
more seriously from the time when I was ill are definitely an issue.

Ahead of me is still some "configuration/detection" that happens
outside of autoconf (mostly in lispbibl.d) as well as the integration
of some gnulib modules (I "started" without any gnulib module, and
added them back as necessary, removing the then superfluous platform
dependent code from CLISP's source files).

What's - at least related to the autotools - still very hacky / not
reliably working is clisp-link.

Re: status update

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-08-16 02:39:14 +0200]:
>
>> Are you prepared to make a release?
>
> Almost, but not yet. Unfortunately.
>

Time is up (pretty much).

Where do we stand now?

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://openvotingconsortium.org http://truepeace.org
http://dhimmi.org http://islamexposedonline.com http://thereligionofpeace.com
Save time: send elected officials to jail directly, bypassing the office.

Re: status update

[Daniel J. <dan...@gm...>]

Ok, here comes the final GSoC status report. Unfortunately, I did not
reach the goal of working (building and passing tests) code on the
major platforms.

I made changes to

 * switch the build system to use the autotools. This is not
   complete. The runtime and memory image get build, as well as the
   clisp "-K dispatcher" executable (all on linux). There's still a
   lot of configuration happening inside the source code (which should
   be done by configure), and not all the required configure tests are
   run (testing for termios, for example). More importantly clisp-link
   is missing support for dynamic module generation and is not yet
   used, thus no acutal linking sets are produced (no base,
   full). Also clisp.h is not yet generated.

 * update the gnulib code used by the runtime. This is halfway done,
   depends mostly on solving the configuration related issues above.

 * update the modules to use up to date gnulib code. This is done for
   rawsock and regexp. They also work with "pre-autotools clisp" and
   also on windows, thus these changes could be used on top of the
   current repo state.

 * update the asdf module. This is done.

 * add a new libmagic module. This is mostly done, but only exports a
   limited subset of the available functions.

 * add new gnulib modules to extend the features of rawsock on windows. I
   had to throw away a lot of code here, the WinAPI is not always
   doing what its documentation claims to do.


To continue the switch to autotools needs to be completed. This means
going through the code, determining what configurations are done
(mostly in lispbibl.d, but also other files and the src/m4 directory)
and add them in as appropriate.


Not reaching the goal of a new release is a big disappointment for me,
but I guess the delay due to my illness was just to much to be able to
complete all the work that needed to be done.


I had a lot of fun working on CLISP, and intend to do so in the
future. Thank you all a lot for your help throughout this summer! :)

(On top of all I'm fighting with an unstable internet connection atm,
I put up the autotools work hopefully in some hours)

Re: status update

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-08-23 20:50:08 +0200]:
>
> Ok, here comes the final GSoC status report. Unfortunately, I did not
> reach the goal of working (building and passing tests) code on the
> major platforms.

Too bad.

Now we need to map out the next steps:

- Merging in your changes: where is your repo, how can I access your
  patchsets &c.

- Making a release. How much effort will you be able to dedicate to
  CLISP during the school year (your studies are, of course, the
  priority)?

> I had a lot of fun working on CLISP, and intend to do so in the
> future. Thank you all a lot for your help throughout this summer! :)

You are welcome!

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://thereligionofpeace.com http://memri.org
http://jihadwatch.org http://dhimmi.org http://americancensorship.org
Don't use force -- get a bigger hammer.

Re: status update

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-08-23 20:50:08 +0200]:
>
> Ok, here comes the final GSoC status report.

Please follow the guidelines in
https://developers.google.com/open-source/gsoc/help/work-product
The upshot is that we need a link to your code.
This is urgent.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://honestreporting.com http://truepeace.org
http://iris.org.il http://camera.org http://openvotingconsortium.org
Professionalism is being dispassionate about your work.

Re: status update

[Daniel J. <dan...@gm...>]

> - Merging in your changes: where is your repo, how can I access your
>  patchsets &c.

Here's my fork of the main repo (I can currently not provide a link
listing only my commits due to the limited internet access):

https://sourceforge.net/u/musteresel/clisp/ci/default/tree/

This contains the changes that are currently useable. One thing I need
to check is whether I included the files related to gnulib (I think I
didn't, just the cache from which they can be reproduced using
gnulib-tool).

Next, the changes towards an autotools based build (and configuration)
system:

https://gist.github.com/musteresel/ee87db0a79e4eb22cfdd2d85afe45bff

This is a squashed commit (I made some changes back and forth which
are not that usefull in a commit history) of the current WIP state,
exported using hg export (thus should be importable using hg import on
the above repository).


Then, the libmagic module:

https://gist.github.com/musteresel/6c3b9eeea10818c7e8f6d6672b030401

It's still missing tests and does not cover the complete functionality
provided by the libmagic on my system.


> The upshot is that we need a link to your code.
> This is urgent.

Sorry for the delay, does the above help?

(A note regarding my limited internet access: This is very
unfortunate, and completely unexpected: My provider -simyo- merged
with/was sold to another company -blau- , which resulted in me being
switched to a different (mobile) network which apparently has a very
weak signal strength or is overloaded.)


> - Making a release. How much effort will you be able to dedicate to
>  CLISP during the school year (your studies are, of course, the
>  priority)?

In September probably almost none (exams). Starting Mid-October I'd
estimate about 10hrs/week, though could also be more (studies start
again but on the other hand then both of my kids are taken care of at
a daycare center; this means a lot more free time.). Next exams are
then around January/February.

Re: status update

[Sam S. <sd...@gn...>]

> * Daniel Jour <qna...@tz...> [2016-08-25 21:35:57 +0200]:
>
>> - Merging in your changes: where is your repo, how can I access your
>>  patchsets &c.
>
> Here's my fork of the main repo (I can currently not provide a link
> listing only my commits due to the limited internet access):
>
> https://sourceforge.net/u/musteresel/clisp/ci/default/tree/
>
> This contains the changes that are currently useable. One thing I need
> to check is whether I included the files related to gnulib (I think I
> didn't, just the cache from which they can be reproduced using
> gnulib-tool).

15613::::Daniel Jour 2016-04-30 clisp-link.in: Replace /bin/pwd with calls to pwd (for NixOS)

pwd is also a shell built-in, so full path is necessary.

15614::::Daniel Jour 2016-04-30 modules/bindings/glibc/linux.lisp: Replace non-standard header include

Fine (I committed an equivalent patch a couple of weeks ago).
This is out of scope though.

15615::::Daniel Jour 2016-05-01 lispbibl.d: don't treat value1 extra if it's not in a register

The metadata (ChangeLog, log summary) is lacking.
I merged this patch with the appropriate metadata.
This is also out of scope though.

15616::::Daniel Jour 2016-05-16 Update asdf to version 3.1.7

Accepted.

15617::::Daniel Jour 2016-05-18 Don't call cat by absolute path

why?

15618::::Daniel Jour 2016-05-18 Fix locale related errors when testing getdate

Accepted.

15619::::Daniel Jour 2016-06-09 Disable TRIVIALMAP_MEMORY and SINGLEMAP_MEMORY on NetBSD

Accepted.

15620::::Daniel Jour 2016-06-19 regexp: use autotools, gnulib, improve buffer handling

0! This is 3 patches in one:
   (a) autotools
   (b) alloca-->static
   (c) do not compute matches when a boolean is returned
1. Why are you replacing stack allocation with arbitrary limits?
2. Is it really necessary to create a separate m4 directory for the
   single file gnulib-cache.m4? It's really ugly! (same for the rawsock
   change below).

15621::::Daniel Jour 2016-06-20 regexp: wrap foreign pointer in REGEX structure
15622::::Daniel Jour 2016-06-21 regexp: Don't return a list when there are many matches
15623::::Daniel Jour 2016-06-21 regexp: Allow selection of return type also for match

these do not apply cleanly without the above.


15624::::Daniel Jour 2016-06-22 Remove comment5 utility, and any remaining non-C comments

Nope.
Each line with separate /* .. */ is ugly.
We need to use either blocks or //.
When did // comments go into the C standard?
Also, I think this should be done together with removing all
pre-processing and renaming all *.d files to *.c.

15625:::tip:Daniel Jour 2016-08-23 rawsock: autotools build system, own gnulib checkout

Copying code from socked.d into rawsock.c is no good.
Also, you removed the configure script, so now people have to install
autoconf to build clisp.
Are you sure this is right?

DIUC that the main change required to make rawsock work on windows was
the switch from rawsock_t to int?



> Next, the changes towards an autotools based build (and configuration)
> system:
>
> https://gist.github.com/musteresel/ee87db0a79e4eb22cfdd2d85afe45bff

what does "wip" in "wip-autotools-export" stand for?
the change to built.d seems incomplete.

> This is a squashed commit (I made some changes back and forth which
> are not that usefull in a commit history) of the current WIP state,
> exported using hg export (thus should be importable using hg import on
> the above repository).

I need to understand what you are doing before importing anything.
The best way would be if you made an isolated change
(e.g., I don't see why you had to touch built.d).


> Then, the libmagic module:
>
> https://gist.github.com/musteresel/6c3b9eeea10818c7e8f6d6672b030401
>
> It's still missing tests and does not cover the complete functionality
> provided by the libmagic on my system.

Okay, by the time you finish it you will probably have write access to
the tree ;-)

>> The upshot is that we need a link to your code.
>> This is urgent.
>
> Sorry for the delay, does the above help?

Yes, thanks.

>> - Making a release. How much effort will you be able to dedicate to
>>  CLISP during the school year (your studies are, of course, the
>>  priority)?
>
> In September probably almost none (exams). Starting Mid-October I'd
> estimate about 10hrs/week, though could also be more ...

Okay, so let us talk then if you are still interested.


-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://palestinefacts.org http://jihadwatch.org
http://dhimmi.org http://think-israel.org http://honestreporting.com
When arguing with the wife, think: do you want to be right or happy?

Re: status update

[Sam S. <sd...@gn...>]

> * Sam Steingold <fq...@ta...> [2016-08-26 14:49:08 -0400]:
>
> 15621::::Daniel Jour 2016-06-20 regexp: wrap foreign pointer in REGEX structure

please take a look at clisp/modules/berkeley-db/bdb.c:bdb_handle() for
getting the regex_t pointer out of the struct.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1404
http://www.childpsy.net/ http://www.dhimmitude.org http://www.memritv.org
http://dhimmi.org http://americancensorship.org http://mideasttruth.com
If you're constantly being mistreated, you're cooperating with the
treatment.

Re: status update

[Daniel J. <dan...@gm...>]

> pwd is also a shell built-in, so full path is necessary.

Hm, I understand why pwd may give (in case of symbolic links) a
different result as a non-builtin pwd. Why do we need the physical
(not logical, i.e. without the symlinks) path, though?

As for /bin/pwd ... this is a bad idea. /bin (the directory) is about
to die (ok, that's a bit drastic, I'm reffering to the "usr merge"
here), and there may be more than one pwd on a single system (this is
the main reason for this change, because it directly affects me. This
could be fixed downstream by the currently few affected distributions,
too. But in the future this might affect every linux distribution that
uses systemd.)

Same goes for /bin/cat: There might be more than one cat, and all of
them might be in a different location than /bin. Thus it's better to
let the user (or the user's configured environment) choose which cat
to use.

Perhaps we could use (if it's still needed then) an autoconf macro to
determine the "correct" cat and pwd? (Using $CAT and $PWD then, so the
user could change these.)



> 15624::::Daniel Jour 2016-06-22 Remove comment5 utility, and any remaining non-C comments
>
> Nope.
> Each line with separate /* .. */ is ugly.
> We need to use either blocks or //.

Hm, indeed ...


> When did // comments go into the C standard?

C99, "ANSI C" (C89/90) didn't have these yet (but I'd bet the major
compilers wouldn't complain when not in -pedantic mode, didn't check
though.)


> Also, I think this should be done together with removing all
> pre-processing and renaming all *.d files to *.c.


This would also extremely simplify the Makefile.am file for
automake. What's the best base (revision) to make such a change on?



> 15620::::Daniel Jour 2016-06-19 regexp: use autotools, gnulib, improve buffer handling
>
> 0! This is 3 patches in one:

Yes, this is too much for a single change. I was working on all three
changes at the same time, which is how this huge commit came into
being (sorry).

> 1. Why are you replacing stack allocation with arbitrary limits?

I don't? I'm switching the buffer size for the error message from
BUFSIZ (which is an unrelated constant that could be easily too much
for the stack) to another constant SAFE_ERROR_MESSAGE_LENGTH which we
can define to a known "good" value (128 is a first start .. if there's
an error message that's longer we can expand this, and consistently
test whether that's too much on the stack or not).

The other change is more important: With a carefully crafted regex
expression one can currently crash (or potentially worse) CLISP,
because the buffer for the matches can overflow the stack. I think
this might be an issue for e.g. web applications that pass a regex
expression such that a - potentially malicious - user can modify
it. As application developer I wouldn't expect an regex of some (arbitrary,
stack frame dependent) length to be able to crush my whole application.

Therefore I changed it so that a small buffer is still allocated on
the stack (because of the speed benefit), but larger buffers get
allocated using dynamic memory allocation (which, in case of a
failure, raises a condition).

It's not waterproof though: A large number of matches (from a
carefully crafted expression) could still cause the LISP stack to
overflow. I don't know how to prevent that, though (except for
enforcing some maximum).


> 2. Is it really necessary to create a separate m4 directory for the
>   single file gnulib-cache.m4? It's really ugly! (same for the
>   rawsock change below).

Yes, that's the directory where all the gnulib (m4 macros) should
reside in, too. Running gnulib-tool --update in the regexp module
directory populates that directory (as well as lib/).

I think the current state (in which I commited this) is unfortunate,
though: The gnulib files (in m4/ and lib/) should be put under (our)
revision control, too. What do you think would be the best approach
here?
(gnulib has some discussion about this very issue:
https://www.gnu.org/software/gnulib/manual/gnulib.html#VCS-Issues)



> 15625:::tip:Daniel Jour 2016-08-23 rawsock: autotools build system, own gnulib checkout
>
> Copying code from socked.d into rawsock.c is no good.

Yes, this is a (dirty) work around: I was not able to update the
gnulib code for core CLISP (thus socket.d).  I was planing to revert
that as soon as core CLISP also uses updated gnulib code.

I'm a bit worried about these dependencies (between the modules -
rawsock needing OS - and the rawsock module and the socket code from
core CLISP) though. Though this is IMO not relevant for now.


> Also, you removed the configure script, so now people have to install
> autoconf to build clisp.
> Are you sure this is right?

I removed it from the repository because it is a generated file
now. It would be part of a source distribution (a release) though,
thus only CLISP developers need to have autoconf/automake/etc.


> DIUC that the main change required to make rawsock work on windows
> was the switch from rawsock_t to int?

Basically, yes. gnulib is (or in part, will be) handling the windows
specific code. rawsock is interfacing to POSIX/BSD sockets, gnulib is
implementing them on windows.

But I noticed that this commit still contains (printf) debugging and
other oddities. I'll fix that ASAP.



> what does "wip" in "wip-autotools-export" stand for?

work in progress. This is the state of switching to autotools at the
end of the coding period. It's not finished or useable yet. Which is
why ...

> the change to built.d seems incomplete.

... a lot of the changes in that commit are indeed still incomplete.


> I need to understand what you are doing before importing anything.
> The best way would be if you made an isolated change
> (e.g., I don't see why you had to touch built.d).

I'll try to isolate the changes needed to get an autotools based build
system from integrating the configuration into autoconf macros. This
should reduce the size of each change. I don't know how fast I'm able
to provide this atm, though.



> please take a look at clisp/modules/berkeley-db/bdb.c:bdb_handle()
> for getting the regex_t pointer out of the struct.

Hm .. is this faster than the approach that I took? It saves a level
of indirection ("one pointer"), right? Or is there something else
wrong with the approach I took in regexp?