clisp-devel — closed list for developers and testers

Re: libffcall and selinux

[Bruno H. <br...@cl...>]

Hi Don,

> I attempted to make various things work with selinux before giving up,

I would be interested to hear more about problems between libffcall
and various security features of Linux distros.

If you have a "How to reproduce" of libffcall running into trouble
(with or without clisp), please report it to libffcall's mailing list:
Distro, customizations/settings of the distro, compilation flags, etc.

By "trouble" I mean something more serious than just a line in a log file.

I'm asking because I tried to reproduce something but couldn't.

Bruno

Queries

[Compro P. <com...@gm...>]

What do the following file naming conventions refer to:
+ spvw*
+ clos*
+ ari*
+ *.d (why not the default *.c)

Should I look into gen* or the generated files?
Can you explain the following line. I was too stuck here but couldn't
understand the syntax from "object" to "=". It would be better if you
can tell me the fine details of why and how do you estimate it lies in
that specific location in memory:
#define S_help_(name)
(((object){.one_o=((oint)((UBYTE*)((&symbol_tab_data.name))+((oint)(tint)((
(1UL<<(2)) ))<<48)))}))

Makefile error

[Compro P. <com...@gm...>]

I updated the source and ran "../configure --without-ffcall
--without-dynamic-modules --with-threads=POSIX_THREADS --with-debug"
(and without "--with-debug" too) then removed -O2 from Makefile and
ran "make" but it had this following error.
gcc -I/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/src
-I/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/build/gllib
-I/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/src/gllib
-g -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
-Wmissing-declarations -Wimplicit -Wno-sign-compare
-Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -pthread
-fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW
-DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DMULTITHREAD
-DPOSIX_THREADS -I. -I. -c ../src/modules.c
if test -d locale; then rm -rf locale; fi
mkdir locale
(cd po && /usr/bin/make && /usr/bin/make install datarootdir=..
localedir='$(datarootdir)/locale' INSTALL_DATA='ln') || (rm -rf locale
; exit 1)
make[1]: Entering directory
'/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/build/po'
make[1]: *** No rule to make target 'sv.gmo', needed by 'all-yes'.  Stop.
make[1]: Leaving directory
'/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/build/po'
make: *** [Makefile:1374: locale] Error 1

How to not build the locale directory? I think it is not necessary.

> 'hg pull' and check that MT build is good now - tested on osx and
> linux-amd64. No need for --enable-portability. There was a problem with
> initialization of special variables bindings.

libffcall and selinux

[<don...@is...>]

 > >I do know that the ffcall tests are affected - something like
 > >executing code from stack.

Here's what I see in audit.log after make check on a machine with
selinux in permissive mode:

type=AVC msg=audit(1495455280.802:2565): avc:  denied  { execheap } for  pid=2256 comm="test1" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1

I attempted to make various things work with selinux before giving up,
so there may be more to report in a default install.

Re: Got my name in the list

[<Joe...@t-...>]

Hi,

Don Cohen wrote:
>I suspect that selinux prevents some tests from working.
>[...]
>I do know that the ffcall tests are affected - something like executing code from stack.

Uh oh. Isn't setting the correct permissions a distribution issue?
Trampolines and similar objects are created when calling back from C to Lisp.

The CLISP FFI testsuite should test these as well (I remember adding such tests), therefore
I'd expect the ffcall tests and CLISP FFI tests to fail or succeed similarly. If only ffcall
tests fail in SELinux, then I'd say some tests are missing from the CLISP FFI testsuite.

Do the failing/crashing tests only involve callbacks or is simply calling C from Lisp already
enough to provoke NX Stack alerts?

Perhaps it would be interesting to see what developers had to change in the runtimes of other
programming languages to not stand in the way of security improvements like stack execution
prevention etc.  I remember reading about such changes, but cannot seem to remember details. :-(

Regards,
	Jörg Höhle

Re: CLISP translated into Swedish...

[Bruno H. <br...@cl...>]

Hi Sebastian,

> how do I interpret these ~S and similar which are interspersed in the
> text strings?

The spec is at
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/sec_22-3.html

> I assume that ~S is for strings

~S is for objects, ~A for strings.

> and that there are others for numbers

Yes.

> and date/time similar to strftime().

This one is not built-in into FORMAT. You'll define it through date-format
in swedish.lisp.

Bruno

Re: CLISP translated into Swedish...

[Bruno H. <br...@cl...>]

Hi Sebastian,

Sebastian Rasmussen wrote:
> I just started to incrementally translate clisp into Swedish.
> I'll upload the translations via translationproject.org, but
> since clisp appears to previously not have had a Swedish
> translation I though I'd send an e-mail about it. :)
> 
> Oh, and if you want me to I can send an additional e-mail
> when I have completed the translation..?

Thanks, Sebastian! This is great.

We do receive notifications about your updates through the Translation Project
(typical subject: "New Swedish PO file for 'clisp'"). I have now committed the
first such copy of sv.po to the clisp repository.

Please continue working this way (don't send your update to this mailing list);
we will continue to take your updates from translationproject.org.

Other than the PO file, in clisp, there's also another file that needs
localization, but doesn't fit in the format of a PO file. If you could tackle
this one as well? Here you see what has been done for previous languages:
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/german.lisp
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/french.lisp
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/russian.lisp
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/dutch.lisp
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/danish.lisp
  https://sourceforge.net/p/clisp/clisp/ci/tip/tree/src/spanish.lisp
For this one, and only this one, I'd like to ask for submission through the
mailing list.

Bruno

New Swedish PO file for 'clisp' (version 2.48-pre1)

[Translation P. R. <ro...@tr...>]

Hello, gentle maintainer.

This is a message from the Translation Project robot.

A revised PO file for textual domain 'clisp' has been submitted
by the Swedish team of translators.  The file is available at:

    http://translationproject.org/latest/clisp/sv.po

(We can arrange things so that in the future such files are automatically
e-mailed to you when they arrive.  Ask at the address below if you want this.)

All other PO files for your package are available in:

    http://translationproject.org/latest/clisp/

Please consider including all of these in your next release, whether
official or a pretest.

Whenever you have a new distribution with a new version number ready,
containing a newer POT file, please send the URL of that distribution
tarball to the address below.  The tarball may be just a pretest or a
snapshot, it does not even have to compile.  It is just used by the
translators when they need some extra translation context.

The following HTML page has been updated:

    http://translationproject.org/domain/clisp.html

If any question arises, please contact the translation coordinator.

Thank you for all your work,

                                The Translation Project robot, in the
                                name of your translation coordinator.
                                <coo...@tr...>

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Just pushed a fix to disable MT/SINGLEMAP_MEMORY combination and tests
should pass now.
As for the sockets tests problem - seems like libffcall presence 'fixes'
it. Will have a deeper look into this later.

On Sat, May 20, 2017 at 4:33 PM, Don Cohen <don-sourceforge-x2012@isis.
cs3-inc.com> wrote:

> Vladimir Tzankov writes:
>  > Compo, Don,
>  >
>  > 'hg pull' and check that MT build is good now - tested on osx and
>  > linux-amd64. No need for --enable-portability. There was a problem with
>  > initialization of special variables bindings.
>
> Thank you!  My nightly MT build worked for the first time since Feb.
>
>  > I'm experiencing check-tests failures on socket tests both on
>  > single and multithread builds. Will investigate further.
>
> Do some of these tests require root?
> I suspect that selinux prevents some tests from working.
> It became enough of a nuisance that I've tended to eventually
> turn it off, so I no longer know exactly what it prevents.
> I do know that the ffcall tests are affected - something like
> executing code from stack.
>

MT & SINGELMAP_MEMORY model

[Vladimir T. <vtz...@gm...>]

Bruno,

I disabled SINGLEMAP_MEMORY model in MT builds. The problem is that this
model assumes single fixed stack address range while every thread requires
it's own stack which is currently malloc-ed. What is the good way to
workaround this? Allocate new threads stacks in the above range?

Besides this I observe weird behaviour with amd64 builds on linux - both
single and multithreaded: without libffcall there is consistent segfault on
sockets tests while with libffcall enabled everything works fine. Will
investigate this further but any clue will be helpful.

--
  vlad

Re: Got my name in the list

[<don...@is...>]

Vladimir Tzankov writes:
 > Compo, Don,
 > 
 > 'hg pull' and check that MT build is good now - tested on osx and
 > linux-amd64. No need for --enable-portability. There was a problem with
 > initialization of special variables bindings.

Thank you!  My nightly MT build worked for the first time since Feb.

 > I'm experiencing check-tests failures on socket tests both on
 > single and multithread builds. Will investigate further.

Do some of these tests require root?
I suspect that selinux prevents some tests from working.
It became enough of a nuisance that I've tended to eventually
turn it off, so I no longer know exactly what it prevents.
I do know that the ffcall tests are affected - something like
executing code from stack.

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

While the build succeeds, the MT tests fail in SPVW_PURE memory model.
SPVW_MIXED passes the tests but I'm not sure it's ok. The problem is
related to per thread special variables initialization - hopefully will fix
it this weekend.

--
  vlad

On Sat, May 20, 2017 at 2:04 AM, Vladimir Tzankov <vtz...@gm...>
wrote:

> Compo, Don,
>
> 'hg pull' and check that MT build is good now - tested on osx and
> linux-amd64. No need for --enable-portability. There was a problem with
> initialization of special variables bindings.
>
> I'm experiencing check-tests failures on socket tests both on single and
> multithread builds. Will investigate further.
>
> --
>  vlad
>
>
>
> On Fri, May 19, 2017 at 8:58 AM, Don Cohen <don-sourceforge-x2012@isis.cs
> 3-inc.com> wrote:
>
>> Compro Prasad writes:
>>
>>  > MT builds successful only with --enable-portability but running
>>  > clisp binary segfaults.
>>
>> More detail would help.
>> This is how my transcript ends:
>>
>>   gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
>> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
>> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
>> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
>> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
>> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
>> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
>> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
>> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
>> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
>> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
>> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>>   ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m
>> 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem)
>> (ext::exit)) (ext::exit t)'
>>   Makefile:1423: recipe for target 'interpreted.mem' failed
>>   make: *** [interpreted.mem] Segmentation fault (core dumped)
>>
>> Is that what happens to you?
>>
>>  > > As I've posted recently, this fails for many of the memory models,
>> inc.
>>  > > whichever is the default on my current system (linux).
>>  > > I suggest starting with Makefile.devel (is that part of the hg
>> source?),
>>
>> Do you have Makefile.devel ?
>>
>>  > > finding the line that says
>>  > >  # Options to pass to configure
>>  > > and changing the next line to
>>  > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
>>  > > storing the result as Makefile.develMT
>>  > > then
>>  > >  make -f Makefile.develMT -k multibuild-porting64-gcc
>>  > > in order to at least find SOME way to build MT.
>>
>> In other words, I hope this is going to show you what other config
>> options you can use to get an mt image to work.
>>
>>
>

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Compo, Don,

'hg pull' and check that MT build is good now - tested on osx and
linux-amd64. No need for --enable-portability. There was a problem with
initialization of special variables bindings.

I'm experiencing check-tests failures on socket tests both on single and
multithread builds. Will investigate further.

--
 vlad



On Fri, May 19, 2017 at 8:58 AM, Don Cohen <don-sourceforge-x2012@isis.
cs3-inc.com> wrote:

> Compro Prasad writes:
>
>  > MT builds successful only with --enable-portability but running
>  > clisp binary segfaults.
>
> More detail would help.
> This is how my transcript ends:
>
>   gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>   ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m
> 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem)
> (ext::exit)) (ext::exit t)'
>   Makefile:1423: recipe for target 'interpreted.mem' failed
>   make: *** [interpreted.mem] Segmentation fault (core dumped)
>
> Is that what happens to you?
>
>  > > As I've posted recently, this fails for many of the memory models,
> inc.
>  > > whichever is the default on my current system (linux).
>  > > I suggest starting with Makefile.devel (is that part of the hg
> source?),
>
> Do you have Makefile.devel ?
>
>  > > finding the line that says
>  > >  # Options to pass to configure
>  > > and changing the next line to
>  > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
>  > > storing the result as Makefile.develMT
>  > > then
>  > >  make -f Makefile.develMT -k multibuild-porting64-gcc
>  > > in order to at least find SOME way to build MT.
>
> In other words, I hope this is going to show you what other config
> options you can use to get an mt image to work.
>
>

Re: Got my name in the list

[Bruno H. <br...@cl...>]

> Are the other flags necessary as pointed by Don Cohen? :
> > gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic ...
> 
> instead my build compiles like this:
> gcc -ggdb  -Wl,--export-dynamic ... 

Here's your problem. The compilation is being done without the various important
flags:
  -DSAFETY=3    - necessary so it doesn't crash soon, for various possible reasons
  -DMULTITHREAD - necessary for the topic on which you want to work
  -DDYNAMIC_FFI - you should better omit this in the first time; it's complexity
                  that you don't need for your work
  and so on.

There are two known good ways to pass CFLAGS:
  - when you invoke configure
  - later, by editing the Makefile.

"make CFLAGS+=-ggdb" is not among the ways I ever tried.

Bruno

Re: Got my name in the list

[<don...@is...>]

Compro Prasad writes:
 > Strikeoff "Giving the path is necessary" on the first line of the previous
 > email.
...
 > > Dropbox/programs/hg/clisp/src/foreign1.lisp ...
 > > *** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"

Actually you don't need FFI in order to work on MT.
It might save trouble to just not include it.

 > > I have Makefile.devel . Does it need attention?
This was (and still is) my suggestion:
> I suggest starting with Makefile.devel (is that part of the hg source?),
> finding the line that says
>  # Options to pass to configure
> and changing the next line to
>  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
> storing the result as Makefile.develMT
> then
>  make -f Makefile.develMT -k multibuild-porting64-gcc
> in order to at least find SOME way to build MT.

This tests a number of different configs, some with enable-portability 
and some without.  I'm hoping that (as happens for me), at least SOME
of these configurations work.
I think you can work on hash tables just as well in any of them, so
you might as well just find one that works and use that.
Even if the hash table code turns out to depend on these config 
differences, you might as well start with one that works.

The result of the make above (eventually) is a whole set of build
directories with names like build-porting64-gcc-debug_gcsafety,
build-porting64-gcc-generational_gc-old_gc, etc.

 

Re: Got my name in the list

[Compro P. <com...@gm...>]

Strikeoff "Giving the path is necessary" on the first line of the previous
email.

On Fri, May 19, 2017 at 2:30 PM, Compro Prasad <com...@gm...>
wrote:

> When I gave the '-ggdb' switch to 'make' the build fails at
> "src/foreign1.lisp" due to absence of FFI. But I have it installed. Giving
> the path is necessary
> Tail of "../configure --enable-portability --with-threads=POSIX_THREADS" :
> Configure findings:
>   FFI:        yes (user requested: default)
>   readline:   yes (user requested: default)
>   libsigsegv: yes
> ./makemake --with-dynamic-ffi  --enable-portability
> --with-threads=POSIX_THREADS > Makefile
> cp -p ../src/cfgunix.lisp config.lisp
> chmod +w config.lisp
> echo '(setq *clhs-root-default* "http://www.ai.mit.edu/
> projects/iiip/doc/CommonLISP/HyperSpec/")' >> config.lisp
>
> To continue building CLISP, the following commands are recommended
>   (cf. unix/INSTALL step 4 ff):
>     /usr/bin/nano config.lisp
>     make
> # CLISP self-test:
>     make check
> # Test non-portable features and modules
>     make extracheck mod-check
> << EOF
>
> Are the other flags necessary as pointed by Don Cohen? :
> > gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>
> instead my build compiles like this:
> gcc -ggdb  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o
> eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o
> array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o
> debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o
> foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -lreadline
> -lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> -lsigsegv  -o lisp.run
>
> And it doesn't stop there. "lisp.run" executes successfully but fails
> after sometime when using "make CFLAGS+=-ggdb" :
> ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW
> -M halfcompiled.mem -q -c ../src/foreign1.lisp -o ./
> ;; Compiling file /mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/
> Dropbox/programs/hg/clisp/src/foreign1.lisp ...
> *** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"
>
> 0 errors, 0 warnings
> make: *** [Makefile:1682: foreign1.fas] Error 1
> << EOF
>
> I have Makefile.devel . Does it need attention?
>
> Again I am pointing out that MT builds using --enable-portability go well
> when using "make" but does segfault when running the clisp binary.
> When not using "--enable-portability" the build goes same as Don Cohen.
> But instead of "make" if "make CFLAGS+=-ggdb" is used the build goes same
> as described above in this email.
>

Re: Got my name in the list

[Compro P. <com...@gm...>]

When I gave the '-ggdb' switch to 'make' the build fails at
"src/foreign1.lisp" due to absence of FFI. But I have it installed. Giving
the path is necessary
Tail of "../configure --enable-portability --with-threads=POSIX_THREADS" :
Configure findings:
  FFI:        yes (user requested: default)
  readline:   yes (user requested: default)
  libsigsegv: yes
./makemake --with-dynamic-ffi  --enable-portability
--with-threads=POSIX_THREADS > Makefile
cp -p ../src/cfgunix.lisp config.lisp
chmod +w config.lisp
echo '(setq *clhs-root-default* "
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/")' >>
config.lisp

To continue building CLISP, the following commands are recommended
  (cf. unix/INSTALL step 4 ff):
    /usr/bin/nano config.lisp
    make
# CLISP self-test:
    make check
# Test non-portable features and modules
    make extracheck mod-check
<< EOF

Are the other flags necessary as pointed by Don Cohen? :
> gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
-Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
-Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
-DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
-DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
-DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
/usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
/usr/local/lib64/libsigsegv.a -lc  -o lisp.run

instead my build compiles like this:
gcc -ggdb  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o
eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o
array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o
debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o
foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -lreadline
-lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
-lsigsegv  -o lisp.run

And it doesn't stop there. "lisp.run" executes successfully but fails after
sometime when using "make CFLAGS+=-ggdb" :
./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW
-M halfcompiled.mem -q -c ../src/foreign1.lisp -o ./
;; Compiling file
/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/src/foreign1.lisp
...
*** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"

0 errors, 0 warnings
make: *** [Makefile:1682: foreign1.fas] Error 1
<< EOF

I have Makefile.devel . Does it need attention?

Again I am pointing out that MT builds using --enable-portability go well
when using "make" but does segfault when running the clisp binary.
When not using "--enable-portability" the build goes same as Don Cohen. But
instead of "make" if "make CFLAGS+=-ggdb" is used the build goes same as
described above in this email.

Re: Got my name in the list

[Bruno H. <br...@cl...>]

Compro Prasad wrote:
> some variables are
> optimized out so its a bit hard to look into it.

To avoid this:
- Use gcc option -ggdb (-g is not as good as -ggdb).
- And don't use -O2, at most -O.

Bruno

Re: Got my name in the list

[<don...@is...>]

Compro Prasad writes:

 > MT builds successful only with --enable-portability but running
 > clisp binary segfaults.

More detail would help.
This is how my transcript ends:

  gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
  ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem) (ext::exit)) (ext::exit t)'
  Makefile:1423: recipe for target 'interpreted.mem' failed
  make: *** [interpreted.mem] Segmentation fault (core dumped)

Is that what happens to you?

 > > As I've posted recently, this fails for many of the memory models, inc.
 > > whichever is the default on my current system (linux).
 > > I suggest starting with Makefile.devel (is that part of the hg source?),

Do you have Makefile.devel ?

 > > finding the line that says
 > >  # Options to pass to configure
 > > and changing the next line to
 > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
 > > storing the result as Makefile.develMT
 > > then
 > >  make -f Makefile.develMT -k multibuild-porting64-gcc
 > > in order to at least find SOME way to build MT.

In other words, I hope this is going to show you what other config
options you can use to get an mt image to work.

Re: Got my name in the list

[Compro P. <com...@gm...>]

MT builds successful only with --enable-portability but running clisp
binary segfaults. Presently debugging the binary to go through the code and
find where is the problem. Do suggest me if there is a better way to debug
as I am using gdb using multiple windows in Emacs and some variables are
optimized out so its a bit hard to look into it. Still don't know how to
look into optimized variables.

On Sat, May 6, 2017 at 5:43 AM, Don Cohen <
don...@is...> wrote:

> Vladimir Tzankov writes:
>  > Hey Compro,
>  >
>  > Let's outline the next steps.
>  > Here is a list off the top of my head - any suggestions and comments
>  > are welcomed.
>  >
>  > * build MT (POSIX_THREADS) and ensure 'check-tests' passes.
>
> Actually, this is likely to be a challenge already in the current state.
> As I've posted recently, this fails for many of the memory models, inc.
> whichever is the default on my current system (linux).
> I suggest starting with Makefile.devel (is that part of the hg source?),
> finding the line that says
>  # Options to pass to configure
> and changing the next line to
>  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
> storing the result as Makefile.develMT
> then
>  make -f Makefile.develMT -k multibuild-porting64-gcc
> in order to at least find SOME way to build MT.
>
> I'd also appreciate if someone could make most or all of the
> memory models build, as they used to only a few months ago.
>
> BTW I have recently run across some fairly reliable ways to cause
> MT bugs, some involving locking and some crashing.
>

Learning more about threading

[Compro P. <com...@gm...>]

Thanks for the advice. I got what you said.
I am sorry, but I had to extend the deadline upto 16th May. I know its too
much but I too don't have any option.
Being out of station I am giving my best at learning more about threading
and lisp in general as I don't have access to any computer.
:-(

On Saturday, May 6, 2017, Vladimir Tzankov <vtz...@gm...> wrote:
> Hey Compro,
>
> Let’s outline the next steps.
> Here is a list off the top of my head - any suggestions and comments
> are welcomed.
>
> * build MT (POSIX_THREADS) and ensure 'check-tests' passes.
> 'check-tests-parallel' won't unless you have a lucky day - removing
> the luck factor is the final goal.
> * get familiar with implementation internals.
> http://clisp.org/impnotes/internals.html is invaluable resource. Also
> this thread is a good place to ask questions. Pay particular attention
> on GC safety issues: http://clisp.org/impnotes/gc.html
> * get familiar with current hash tables implementation.
> * write tests causing segfaults or infinite loops involving hash
> tables and study the reasons (e.g. one such ‘test’ is:
>
> (defvar *ht* (make-hash-table :test #'eql))
> (defun ht-thread ()
>  (loop
>     (setf (gethash (random 100) *ht*) (random 100))
>     (remhash (random 100) *ht*)))
> (loop repeat 10 do (make-thread #'ht-thread))
>
> this will segfault shortly after threads go alive).
>
>
> It will be easier to plan how to attack the problems, once you have
> understanding what’s going on
>
> BR
>   Vlad
>
> On Thu, May 4, 2017 at 10:13 PM, Compro Prasad <com...@gm...>
wrote:
>> Tomorrow is a busy schedule for me. The day after tomorrow I may be
>> confirming about my return. Most probably it will be not after 10 May.
So, I
>> am fixing my return deadline on 10 May 2017.
>>
>> Thank you again for my proposal approval.
>

Re: Got my name in the list

[<don...@is...>]

Vladimir Tzankov writes:
 > Hey Compro,
 > 
 > Let's outline the next steps.
 > Here is a list off the top of my head - any suggestions and comments
 > are welcomed.
 > 
 > * build MT (POSIX_THREADS) and ensure 'check-tests' passes.

Actually, this is likely to be a challenge already in the current state.
As I've posted recently, this fails for many of the memory models, inc.
whichever is the default on my current system (linux).
I suggest starting with Makefile.devel (is that part of the hg source?),
finding the line that says 
 # Options to pass to configure
and changing the next line to 
 MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
storing the result as Makefile.develMT
then
 make -f Makefile.develMT -k multibuild-porting64-gcc
in order to at least find SOME way to build MT.

I'd also appreciate if someone could make most or all of the
memory models build, as they used to only a few months ago.

BTW I have recently run across some fairly reliable ways to cause
MT bugs, some involving locking and some crashing.

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Hey Compro,

Let’s outline the next steps.
Here is a list off the top of my head - any suggestions and comments
are welcomed.

* build MT (POSIX_THREADS) and ensure 'check-tests' passes.
'check-tests-parallel' won't unless you have a lucky day - removing
the luck factor is the final goal.
* get familiar with implementation internals.
http://clisp.org/impnotes/internals.html is invaluable resource. Also
this thread is a good place to ask questions. Pay particular attention
on GC safety issues: http://clisp.org/impnotes/gc.html
* get familiar with current hash tables implementation.
* write tests causing segfaults or infinite loops involving hash
tables and study the reasons (e.g. one such ‘test’ is:

(defvar *ht* (make-hash-table :test #'eql))
(defun ht-thread ()
 (loop
    (setf (gethash (random 100) *ht*) (random 100))
    (remhash (random 100) *ht*)))
(loop repeat 10 do (make-thread #'ht-thread))

this will segfault shortly after threads go alive).


It will be easier to plan how to attack the problems, once you have
understanding what’s going on

BR
  Vlad

On Thu, May 4, 2017 at 10:13 PM, Compro Prasad <com...@gm...> wrote:
> Tomorrow is a busy schedule for me. The day after tomorrow I may be
> confirming about my return. Most probably it will be not after 10 May. So, I
> am fixing my return deadline on 10 May 2017.
>
> Thank you again for my proposal approval.

Got my name in the list

[Compro P. <com...@gm...>]

Tomorrow is a busy schedule for me. The day after tomorrow I may be
confirming about my return. Most probably it will be not after 10 May. So,
I am fixing my return deadline on 10 May 2017.

Thank you again for my proposal approval.

Re: GSoC 2017: we have a slot

[Sam S. <sd...@gn...>]

> * Compro Prasad <pbzcebcenfnq@tznvy.pbz> [2017-04-26 17:20:06 +0500]:
>
> Will there be a specific mentor or I can get help from anybody or its
> both the ways?

Vladimir is the primary mentor - he knows way more about MT.
Please keep all communication on this list (clisp-devel).

> If its positive then I may cancel the travel else I will try joining you
> after 15th May or after my vacations(July).
> https://developers.google.com/open-source/gsoc/timeline

I am not sure what you mean by "joining us", but the next critical date
on the timeline is June 26.
By this date you need to have most of the work either done or in a state
where we will be confident that you _will_ finish it on time.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://honestreporting.com
https://jihadwatch.org http://www.dhimmitude.org http://jij.org
Marriage is the sole cause of divorce.