clisp-devel

Got my name in the list

[Compro P. <com...@gm...>]

Tomorrow is a busy schedule for me. The day after tomorrow I may be
confirming about my return. Most probably it will be not after 10 May. So,
I am fixing my return deadline on 10 May 2017.

Thank you again for my proposal approval.

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Hey Compro,

Let’s outline the next steps.
Here is a list off the top of my head - any suggestions and comments
are welcomed.

* build MT (POSIX_THREADS) and ensure 'check-tests' passes.
'check-tests-parallel' won't unless you have a lucky day - removing
the luck factor is the final goal.
* get familiar with implementation internals.
http://clisp.org/impnotes/internals.html is invaluable resource. Also
this thread is a good place to ask questions. Pay particular attention
on GC safety issues: http://clisp.org/impnotes/gc.html
* get familiar with current hash tables implementation.
* write tests causing segfaults or infinite loops involving hash
tables and study the reasons (e.g. one such ‘test’ is:

(defvar *ht* (make-hash-table :test #'eql))
(defun ht-thread ()
 (loop
    (setf (gethash (random 100) *ht*) (random 100))
    (remhash (random 100) *ht*)))
(loop repeat 10 do (make-thread #'ht-thread))

this will segfault shortly after threads go alive).


It will be easier to plan how to attack the problems, once you have
understanding what’s going on

BR
  Vlad

On Thu, May 4, 2017 at 10:13 PM, Compro Prasad <com...@gm...> wrote:
> Tomorrow is a busy schedule for me. The day after tomorrow I may be
> confirming about my return. Most probably it will be not after 10 May. So, I
> am fixing my return deadline on 10 May 2017.
>
> Thank you again for my proposal approval.

Re: Got my name in the list

[<don...@is...>]

Vladimir Tzankov writes:
 > Hey Compro,
 > 
 > Let's outline the next steps.
 > Here is a list off the top of my head - any suggestions and comments
 > are welcomed.
 > 
 > * build MT (POSIX_THREADS) and ensure 'check-tests' passes.

Actually, this is likely to be a challenge already in the current state.
As I've posted recently, this fails for many of the memory models, inc.
whichever is the default on my current system (linux).
I suggest starting with Makefile.devel (is that part of the hg source?),
finding the line that says 
 # Options to pass to configure
and changing the next line to 
 MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
storing the result as Makefile.develMT
then
 make -f Makefile.develMT -k multibuild-porting64-gcc
in order to at least find SOME way to build MT.

I'd also appreciate if someone could make most or all of the
memory models build, as they used to only a few months ago.

BTW I have recently run across some fairly reliable ways to cause
MT bugs, some involving locking and some crashing.

Re: Got my name in the list

[Compro P. <com...@gm...>]

MT builds successful only with --enable-portability but running clisp
binary segfaults. Presently debugging the binary to go through the code and
find where is the problem. Do suggest me if there is a better way to debug
as I am using gdb using multiple windows in Emacs and some variables are
optimized out so its a bit hard to look into it. Still don't know how to
look into optimized variables.

On Sat, May 6, 2017 at 5:43 AM, Don Cohen <
don...@is...> wrote:

> Vladimir Tzankov writes:
>  > Hey Compro,
>  >
>  > Let's outline the next steps.
>  > Here is a list off the top of my head - any suggestions and comments
>  > are welcomed.
>  >
>  > * build MT (POSIX_THREADS) and ensure 'check-tests' passes.
>
> Actually, this is likely to be a challenge already in the current state.
> As I've posted recently, this fails for many of the memory models, inc.
> whichever is the default on my current system (linux).
> I suggest starting with Makefile.devel (is that part of the hg source?),
> finding the line that says
>  # Options to pass to configure
> and changing the next line to
>  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
> storing the result as Makefile.develMT
> then
>  make -f Makefile.develMT -k multibuild-porting64-gcc
> in order to at least find SOME way to build MT.
>
> I'd also appreciate if someone could make most or all of the
> memory models build, as they used to only a few months ago.
>
> BTW I have recently run across some fairly reliable ways to cause
> MT bugs, some involving locking and some crashing.
>

Re: Got my name in the list

[<don...@is...>]

Compro Prasad writes:

 > MT builds successful only with --enable-portability but running
 > clisp binary segfaults.

More detail would help.
This is how my transcript ends:

  gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
  ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem) (ext::exit)) (ext::exit t)'
  Makefile:1423: recipe for target 'interpreted.mem' failed
  make: *** [interpreted.mem] Segmentation fault (core dumped)

Is that what happens to you?

 > > As I've posted recently, this fails for many of the memory models, inc.
 > > whichever is the default on my current system (linux).
 > > I suggest starting with Makefile.devel (is that part of the hg source?),

Do you have Makefile.devel ?

 > > finding the line that says
 > >  # Options to pass to configure
 > > and changing the next line to
 > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
 > > storing the result as Makefile.develMT
 > > then
 > >  make -f Makefile.develMT -k multibuild-porting64-gcc
 > > in order to at least find SOME way to build MT.

In other words, I hope this is going to show you what other config
options you can use to get an mt image to work.

Re: Got my name in the list

[Bruno H. <br...@cl...>]

Compro Prasad wrote:
> some variables are
> optimized out so its a bit hard to look into it.

To avoid this:
- Use gcc option -ggdb (-g is not as good as -ggdb).
- And don't use -O2, at most -O.

Bruno

Re: Got my name in the list

[Compro P. <com...@gm...>]

When I gave the '-ggdb' switch to 'make' the build fails at
"src/foreign1.lisp" due to absence of FFI. But I have it installed. Giving
the path is necessary
Tail of "../configure --enable-portability --with-threads=POSIX_THREADS" :
Configure findings:
  FFI:        yes (user requested: default)
  readline:   yes (user requested: default)
  libsigsegv: yes
./makemake --with-dynamic-ffi  --enable-portability
--with-threads=POSIX_THREADS > Makefile
cp -p ../src/cfgunix.lisp config.lisp
chmod +w config.lisp
echo '(setq *clhs-root-default* "
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/")' >>
config.lisp

To continue building CLISP, the following commands are recommended
  (cf. unix/INSTALL step 4 ff):
    /usr/bin/nano config.lisp
    make
# CLISP self-test:
    make check
# Test non-portable features and modules
    make extracheck mod-check
<< EOF

Are the other flags necessary as pointed by Don Cohen? :
> gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
-Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
-Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
-DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
-DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
-DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
/usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
/usr/local/lib64/libsigsegv.a -lc  -o lisp.run

instead my build compiles like this:
gcc -ggdb  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o
eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o
array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o
debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o
foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -lreadline
-lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
-lsigsegv  -o lisp.run

And it doesn't stop there. "lisp.run" executes successfully but fails after
sometime when using "make CFLAGS+=-ggdb" :
./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW
-M halfcompiled.mem -q -c ../src/foreign1.lisp -o ./
;; Compiling file
/mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/Dropbox/programs/hg/clisp/src/foreign1.lisp
...
*** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"

0 errors, 0 warnings
make: *** [Makefile:1682: foreign1.fas] Error 1
<< EOF

I have Makefile.devel . Does it need attention?

Again I am pointing out that MT builds using --enable-portability go well
when using "make" but does segfault when running the clisp binary.
When not using "--enable-portability" the build goes same as Don Cohen. But
instead of "make" if "make CFLAGS+=-ggdb" is used the build goes same as
described above in this email.

Re: Got my name in the list

[Bruno H. <br...@cl...>]

> Are the other flags necessary as pointed by Don Cohen? :
> > gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic ...
> 
> instead my build compiles like this:
> gcc -ggdb  -Wl,--export-dynamic ... 

Here's your problem. The compilation is being done without the various important
flags:
  -DSAFETY=3    - necessary so it doesn't crash soon, for various possible reasons
  -DMULTITHREAD - necessary for the topic on which you want to work
  -DDYNAMIC_FFI - you should better omit this in the first time; it's complexity
                  that you don't need for your work
  and so on.

There are two known good ways to pass CFLAGS:
  - when you invoke configure
  - later, by editing the Makefile.

"make CFLAGS+=-ggdb" is not among the ways I ever tried.

Bruno

Re: Got my name in the list

[Compro P. <com...@gm...>]

Strikeoff "Giving the path is necessary" on the first line of the previous
email.

On Fri, May 19, 2017 at 2:30 PM, Compro Prasad <com...@gm...>
wrote:

> When I gave the '-ggdb' switch to 'make' the build fails at
> "src/foreign1.lisp" due to absence of FFI. But I have it installed. Giving
> the path is necessary
> Tail of "../configure --enable-portability --with-threads=POSIX_THREADS" :
> Configure findings:
>   FFI:        yes (user requested: default)
>   readline:   yes (user requested: default)
>   libsigsegv: yes
> ./makemake --with-dynamic-ffi  --enable-portability
> --with-threads=POSIX_THREADS > Makefile
> cp -p ../src/cfgunix.lisp config.lisp
> chmod +w config.lisp
> echo '(setq *clhs-root-default* "http://www.ai.mit.edu/
> projects/iiip/doc/CommonLISP/HyperSpec/")' >> config.lisp
>
> To continue building CLISP, the following commands are recommended
>   (cf. unix/INSTALL step 4 ff):
>     /usr/bin/nano config.lisp
>     make
> # CLISP self-test:
>     make check
> # Test non-portable features and modules
>     make extracheck mod-check
> << EOF
>
> Are the other flags necessary as pointed by Don Cohen? :
> > gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>
> instead my build compiles like this:
> gcc -ggdb  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o spvwtabo.o
> eval.o control.o encoding.o pathname.o stream.o socket.o io.o funarg.o
> array.o hashtabl.o list.o package.o record.o weak.o sequence.o charstrg.o
> debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o i18n.o
> foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -lreadline
> -lncurses -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> -lsigsegv  -o lisp.run
>
> And it doesn't stop there. "lisp.run" executes successfully but fails
> after sometime when using "make CFLAGS+=-ggdb" :
> ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m 2MW
> -M halfcompiled.mem -q -c ../src/foreign1.lisp -o ./
> ;; Compiling file /mnt/eb7b593c-d20e-4393-be34-3b108921f50e/home/compro/
> Dropbox/programs/hg/clisp/src/foreign1.lisp ...
> *** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"
>
> 0 errors, 0 warnings
> make: *** [Makefile:1682: foreign1.fas] Error 1
> << EOF
>
> I have Makefile.devel . Does it need attention?
>
> Again I am pointing out that MT builds using --enable-portability go well
> when using "make" but does segfault when running the clisp binary.
> When not using "--enable-portability" the build goes same as Don Cohen.
> But instead of "make" if "make CFLAGS+=-ggdb" is used the build goes same
> as described above in this email.
>

Re: Got my name in the list

[<don...@is...>]

Compro Prasad writes:
 > Strikeoff "Giving the path is necessary" on the first line of the previous
 > email.
...
 > > Dropbox/programs/hg/clisp/src/foreign1.lisp ...
 > > *** - SYSTEM::%FIND-PACKAGE: There is no package with name "FFI"

Actually you don't need FFI in order to work on MT.
It might save trouble to just not include it.

 > > I have Makefile.devel . Does it need attention?
This was (and still is) my suggestion:
> I suggest starting with Makefile.devel (is that part of the hg source?),
> finding the line that says
>  # Options to pass to configure
> and changing the next line to
>  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
> storing the result as Makefile.develMT
> then
>  make -f Makefile.develMT -k multibuild-porting64-gcc
> in order to at least find SOME way to build MT.

This tests a number of different configs, some with enable-portability 
and some without.  I'm hoping that (as happens for me), at least SOME
of these configurations work.
I think you can work on hash tables just as well in any of them, so
you might as well just find one that works and use that.
Even if the hash table code turns out to depend on these config 
differences, you might as well start with one that works.

The result of the make above (eventually) is a whole set of build
directories with names like build-porting64-gcc-debug_gcsafety,
build-porting64-gcc-generational_gc-old_gc, etc.

 

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Compo, Don,

'hg pull' and check that MT build is good now - tested on osx and
linux-amd64. No need for --enable-portability. There was a problem with
initialization of special variables bindings.

I'm experiencing check-tests failures on socket tests both on single and
multithread builds. Will investigate further.

--
 vlad



On Fri, May 19, 2017 at 8:58 AM, Don Cohen <don-sourceforge-x2012@isis.
cs3-inc.com> wrote:

> Compro Prasad writes:
>
>  > MT builds successful only with --enable-portability but running
>  > clisp binary segfaults.
>
> More detail would help.
> This is how my transcript ends:
>
>   gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>   ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m
> 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem)
> (ext::exit)) (ext::exit t)'
>   Makefile:1423: recipe for target 'interpreted.mem' failed
>   make: *** [interpreted.mem] Segmentation fault (core dumped)
>
> Is that what happens to you?
>
>  > > As I've posted recently, this fails for many of the memory models,
> inc.
>  > > whichever is the default on my current system (linux).
>  > > I suggest starting with Makefile.devel (is that part of the hg
> source?),
>
> Do you have Makefile.devel ?
>
>  > > finding the line that says
>  > >  # Options to pass to configure
>  > > and changing the next line to
>  > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
>  > > storing the result as Makefile.develMT
>  > > then
>  > >  make -f Makefile.develMT -k multibuild-porting64-gcc
>  > > in order to at least find SOME way to build MT.
>
> In other words, I hope this is going to show you what other config
> options you can use to get an mt image to work.
>
>

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

While the build succeeds, the MT tests fail in SPVW_PURE memory model.
SPVW_MIXED passes the tests but I'm not sure it's ok. The problem is
related to per thread special variables initialization - hopefully will fix
it this weekend.

--
  vlad

On Sat, May 20, 2017 at 2:04 AM, Vladimir Tzankov <vtz...@gm...>
wrote:

> Compo, Don,
>
> 'hg pull' and check that MT build is good now - tested on osx and
> linux-amd64. No need for --enable-portability. There was a problem with
> initialization of special variables bindings.
>
> I'm experiencing check-tests failures on socket tests both on single and
> multithread builds. Will investigate further.
>
> --
>  vlad
>
>
>
> On Fri, May 19, 2017 at 8:58 AM, Don Cohen <don-sourceforge-x2012@isis.cs
> 3-inc.com> wrote:
>
>> Compro Prasad writes:
>>
>>  > MT builds successful only with --enable-portability but running
>>  > clisp binary segfaults.
>>
>> More detail would help.
>> This is how my transcript ends:
>>
>>   gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type
>> -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral
>> -Wno-shift-negative-value -fwrapv -pthread -fno-strict-aliasing -ggdb -O0
>> -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE
>> -DMULTITHREAD -DPOSIX_THREADS -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI
>> -DDYNAMIC_MODULES  -Wl,--export-dynamic spvw.o spvwtabf.o spvwtabs.o
>> spvwtabo.o eval.o control.o encoding.o pathname.o stream.o socket.o io.o
>> funarg.o array.o hashtabl.o list.o package.o record.o weak.o sequence.o
>> charstrg.o debug.o error.o misc.o time.o predtype.o symbol.o lisparit.o
>> i18n.o foreign.o unixaux.o zthread.o built.o modules.o libgnu.a  -ldl
>> /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a
>> /usr/local/lib64/libsigsegv.a -lc  -o lisp.run
>>   ./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -m
>> 2MW -lp ../src/ -x '(and (load "../src/init.lisp") (sys::%saveinitmem)
>> (ext::exit)) (ext::exit t)'
>>   Makefile:1423: recipe for target 'interpreted.mem' failed
>>   make: *** [interpreted.mem] Segmentation fault (core dumped)
>>
>> Is that what happens to you?
>>
>>  > > As I've posted recently, this fails for many of the memory models,
>> inc.
>>  > > whichever is the default on my current system (linux).
>>  > > I suggest starting with Makefile.devel (is that part of the hg
>> source?),
>>
>> Do you have Makefile.devel ?
>>
>>  > > finding the line that says
>>  > >  # Options to pass to configure
>>  > > and changing the next line to
>>  > >  MULTIBUILD_OPTIONS =  --with-threads=POSIX_THREADS
>>  > > storing the result as Makefile.develMT
>>  > > then
>>  > >  make -f Makefile.develMT -k multibuild-porting64-gcc
>>  > > in order to at least find SOME way to build MT.
>>
>> In other words, I hope this is going to show you what other config
>> options you can use to get an mt image to work.
>>
>>
>

Re: Got my name in the list

[<don...@is...>]

Vladimir Tzankov writes:
 > Compo, Don,
 > 
 > 'hg pull' and check that MT build is good now - tested on osx and
 > linux-amd64. No need for --enable-portability. There was a problem with
 > initialization of special variables bindings.

Thank you!  My nightly MT build worked for the first time since Feb.

 > I'm experiencing check-tests failures on socket tests both on
 > single and multithread builds. Will investigate further.

Do some of these tests require root?
I suspect that selinux prevents some tests from working.
It became enough of a nuisance that I've tended to eventually
turn it off, so I no longer know exactly what it prevents.
I do know that the ffcall tests are affected - something like
executing code from stack.

Re: Got my name in the list

[Vladimir T. <vtz...@gm...>]

Just pushed a fix to disable MT/SINGLEMAP_MEMORY combination and tests
should pass now.
As for the sockets tests problem - seems like libffcall presence 'fixes'
it. Will have a deeper look into this later.

On Sat, May 20, 2017 at 4:33 PM, Don Cohen <don-sourceforge-x2012@isis.
cs3-inc.com> wrote:

> Vladimir Tzankov writes:
>  > Compo, Don,
>  >
>  > 'hg pull' and check that MT build is good now - tested on osx and
>  > linux-amd64. No need for --enable-portability. There was a problem with
>  > initialization of special variables bindings.
>
> Thank you!  My nightly MT build worked for the first time since Feb.
>
>  > I'm experiencing check-tests failures on socket tests both on
>  > single and multithread builds. Will investigate further.
>
> Do some of these tests require root?
> I suspect that selinux prevents some tests from working.
> It became enough of a nuisance that I've tended to eventually
> turn it off, so I no longer know exactly what it prevents.
> I do know that the ffcall tests are affected - something like
> executing code from stack.
>

Re: Got my name in the list

[<Joe...@t-...>]

Hi,

Don Cohen wrote:
>I suspect that selinux prevents some tests from working.
>[...]
>I do know that the ffcall tests are affected - something like executing code from stack.

Uh oh. Isn't setting the correct permissions a distribution issue?
Trampolines and similar objects are created when calling back from C to Lisp.

The CLISP FFI testsuite should test these as well (I remember adding such tests), therefore
I'd expect the ffcall tests and CLISP FFI tests to fail or succeed similarly. If only ffcall
tests fail in SELinux, then I'd say some tests are missing from the CLISP FFI testsuite.

Do the failing/crashing tests only involve callbacks or is simply calling C from Lisp already
enough to provoke NX Stack alerts?

Perhaps it would be interesting to see what developers had to change in the runtimes of other
programming languages to not stand in the way of security improvements like stack execution
prevention etc.  I remember reading about such changes, but cannot seem to remember details. :-(

Regards,
	Jörg Höhle

libffcall and selinux

[<don...@is...>]

 > >I do know that the ffcall tests are affected - something like
 > >executing code from stack.

Here's what I see in audit.log after make check on a machine with
selinux in permissive mode:

type=AVC msg=audit(1495455280.802:2565): avc:  denied  { execheap } for  pid=2256 comm="test1" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=1

I attempted to make various things work with selinux before giving up,
so there may be more to report in a default install.

Re: libffcall and selinux

[Bruno H. <br...@cl...>]

Hi Don,

> I attempted to make various things work with selinux before giving up,

I would be interested to hear more about problems between libffcall
and various security features of Linux distros.

If you have a "How to reproduce" of libffcall running into trouble
(with or without clisp), please report it to libffcall's mailing list:
Distro, customizations/settings of the distro, compilation flags, etc.

By "trouble" I mean something more serious than just a line in a log file.

I'm asking because I tried to reproduce something but couldn't.

Bruno

Re: libffcall and selinux

[<don...@is...>]

Bruno Haible writes:
 > Hi Don,
 > 
 > > I attempted to make various things work with selinux before giving up,
 > 
 > I would be interested to hear more about problems between libffcall
 > and various security features of Linux distros.
 > 
 > If you have a "How to reproduce" of libffcall running into trouble
 > (with or without clisp), please report it to libffcall's mailing list:
 > Distro, customizations/settings of the distro, compilation flags, etc.
 > 
 > By "trouble" I mean something more serious than just a line in a log file.
 > 
 > I'm asking because I tried to reproduce something but couldn't.
 > 
 > Bruno
 > 

Here's my procedure:
dowloaded from https://getfedora.org/en/server/download/
Fedora-Server-dvd-x86_64-25-1.3.iso (2GB)
start a virtual machine (using virtual machine manager in linux,
but this shouldn't matter), install above (create user don to
do builds)
login as root at console to find ip addr, then ssh to it as root.

 yum update

[now that I think of it, this probably just makes things less repeatable
it also takes a long time, so probably should have skipped it]

 yum install emacs emacs-el gcc ncurses ncurses-devel ncurses-term
 ncurses-libs readline readline-devel

I think all the ncurses stuff us needed for readline,
gcc needed for building stuff below, 
emacs needed just to live, also good for collecting transcripts

Now I run emacs, start a shell buffer with the intent (frustrated
in part as you see below) of using that buffer as a transcript
for the rest.

I thought I'd save a little time (not clear I did) by scp'ing my
hg directory to the vm.

Interestingly at this point I already see a bunch of AVC denied messages
in /var/log/audit/audit.log

But at least I can see clisp/unix/INSTALL which says to get ligsigsegv from 
http://sourceforge.net/projects/libsigsegv 
but that says it's moved to http://savannah.gnu.org/projects/libsigsegv/
(maybe you should update unix/INSTALL ?)

I use the browser of the host machine to get 
http://ftp.gnu.org/gnu/libsigsegv/libsigsegv-2.11.tar.gz
and scp that to the VM (in /root)

gunzip, tar, cd, ./configure, make, make check, make install all
seem to go smoothly, audit.log shows no additional avc's.

unix install then says https://www.gnu.org/software/libffcall/
and that points me to
 https://haible.de/bruno/gnu/libffcall-1.13-20170225.tar.gz
which I again scp to the vm

Now gunzip,tar, read instructions and do mkdir, cd, ../configure

At this point I see a new AVC in audit.log:

type=AVC msg=audit(1495493082.183:494): avc:  denied  { execheap } for  pid=3540 comm="conftest" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0

The end of the configure shows:
  checking for working shared memory... (cached) yes
  checking for sys/sysmacros.h... (cached) yes
  checking whether code in malloc()ed memory is executable... (cached) no
  checking whether mprotect can make malloc()ed memory executable... (cached) no
  checking whether mprotect can make mmap()ed memory executable... (cached) yes
  configure: creating ./config.status
  config.status: creating Makefile
  config.status: creating config.h
  config.status: creating trampoline_r.h
  config.status: executing libtool commands
  [root@localhost builddir]# 

This is probably what you expect.
I now try make - no new AVC.
then make check - still no new AVC -- which surprises me. 
I thought the last time I tried make check it did result in AVC's.
then make install, still no avc

So now I try to build clisp:
su don, cd to hg/clisp, hg pull - hg not yet installed, back out, install,
su don, cd to hg/clisp, hg pull
(1 changeset, 4 changes to 4 files, which I hope fixed swedish problem
that prevented last night's build)

Not sure why, but this seems to be my traditional configure:
./configure CC='gcc -m64' --with-libsigsegv-prefix=/usr/local --disable-maintainer-mode --with-debug --with-module=rawsock build-dir

no new avc's
configure says FFI yes, libsigsegv yes, readline no - so I guess I had
to install something else to get that (any suggestions?)
But this was not meant as a test of readline so I continue with make

It ends up with this - looks like a problem remaining from recent
addition of swedish

gcc -m64 -I/home/don/hg/clisp/src -I/home/don/hg/clisp/build-dir/gllib -I/home/don/hg/clisp/src/gllib -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DDYNAMIC_MODULES -I. -I. -c ../src/modules.c
if test -d locale; then rm -rf locale; fi
mkdir locale
(cd po && make && make install datarootdir=.. localedir='$(datarootdir)/locale' INSTALL_DATA='ln') || (rm -rf locale ; exit 1)
make[1]: Entering directory '/home/don/hg/clisp/build-dir/po'
make[1]: *** No rule to make target 'sv.gmo', needed by 'all-yes'.  Stop.
make[1]: Leaving directory '/home/don/hg/clisp/build-dir/po'
Makefile:1382: recipe for target 'locale' failed
make: *** [locale] Error 1
[don@localhost build-dir]$ 

I'll leave this vm running (but idle) in hope of an update that will
allow the build to continue.

Re: libffcall and selinux

[Bruno H. <br...@cl...>]

Hi Don,

> Now gunzip,tar, read instructions and do mkdir, cd, ../configure
> 
> At this point I see a new AVC in audit.log:
> 
> type=AVC msg=audit(1495493082.183:494): avc:  denied  { execheap } for  pid=3540 comm="conftest" scontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tcontext=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 tclass=process permissive=0
> 
> The end of the configure shows:
>   checking for working shared memory... (cached) yes
>   checking for sys/sysmacros.h... (cached) yes
>   checking whether code in malloc()ed memory is executable... (cached) no
>   checking whether mprotect can make malloc()ed memory executable... (cached) no
>   checking whether mprotect can make mmap()ed memory executable... (cached) yes
>   configure: creating ./config.status
>   config.status: creating Makefile
>   config.status: creating config.h
>   config.status: creating trampoline_r.h
>   config.status: executing libtool commands
>   [root@localhost builddir]# 
> 
> This is probably what you expect.
> I now try make - no new AVC.
> then make check - still no new AVC -- which surprises me. 

Good. This means that the workaround for Linux+PaX or SELinux kernels and for
HardenedBSD, that I added to libffcall on 2017-01-29, is working fine.

1 log message during configuration is acceptable, I think.

> (maybe you should update unix/INSTALL ?)

Done.

> Not sure why, but this seems to be my traditional configure:
> ./configure CC='gcc -m64' --with-libsigsegv-prefix=/usr/local
>   --disable-maintainer-mode --with-debug --with-module=rawsock build-dir

Ouch. Don't install libraries in /usr/local *EVER*. You'll invariably
get trapped by a decades-old GCC bug if you do so. /usr/local is ONLY good
for executables. Any other --prefix value is OK; only /usr/local is so bad.

Bruno

Re: libffcall and selinux

[<don...@is...>]

 > > Not sure why, but this seems to be my traditional configure:
 > > ./configure CC='gcc -m64' --with-libsigsegv-prefix=/usr/local
 > >   --disable-maintainer-mode --with-debug --with-module=rawsock build-dir
 > 
 > Ouch. Don't install libraries in /usr/local *EVER*. You'll invariably
 > get trapped by a decades-old GCC bug if you do so. /usr/local is ONLY good
 > for executables. Any other --prefix value is OK; only /usr/local is so bad.

And yet that's where the library ends up when I do plain make install
as the directions say.

However, after hg pull retrieves a few new changes I try without
that and it still finds libsigsegv, and now also finds readline too.

But then make still ends with this:

gcc -m64 -I/home/don/hg/clisp/src -I/home/don/hg/clisp/build-dir/gllib -I/home/don/hg/clisp/src/gllib -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -fwrapv -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES -I. -I. -c ../src/modules.c
if test -d locale; then rm -rf locale; fi
mkdir locale
(cd po && make && make install datarootdir=.. localedir='$(datarootdir)/locale' INSTALL_DATA='ln') || (rm -rf locale ; exit 1)
make[1]: Entering directory '/home/don/hg/clisp/build-dir/po'
make[1]: *** No rule to make target 'sv.gmo', needed by 'all-yes'.  Stop.
make[1]: Leaving directory '/home/don/hg/clisp/build-dir/po'
Makefile:1382: recipe for target 'locale' failed
make: *** [locale] Error 1
[don@localhost build-dir]$ 

Re: libffcall and selinux

[<don...@is...>]

  mkdir locale
  (cd po && make && make install datarootdir=.. localedir='$(datarootdir)/locale' INSTALL_DATA='ln') || (rm -rf locale ; exit 1)
  make[1]: Entering directory '/home/don/hg/clisp/build-dir/po'
  make[1]: *** No rule to make target 'sv.gmo', needed by 'all-yes'.  Stop.
  make[1]: Leaving directory '/home/don/hg/clisp/build-dir/po'
  Makefile:1382: recipe for target 'locale' failed
  make: *** [locale] Error 1
  [don@localhost build-dir]$ 

I don't understand why this build is different from the one on the
host machine, where make reports nothing to be done.
But since that's not the main point of this exercise, I just copy 
sc.gmo and then clisplow_sv.gmo from the host to the VM and then 
make seems to finish ok.
Then make check ends with:

EQL-OK: NIL
(PROGN (SETQ *SERVER* (SHOW (SOCKET-SERVER 9090)) *SOCKET-1* (SHOW (SOCKET-CONNECT 9090 "localhost" :TIMEOUT 0 :BUFFERED NIL)) *SOCKET-2* (SHOW (SOCKET-ACCEPT *SERVER* :BUFFERED NIL))) (WRITE-CHAR #\a *SOCKET-1*))

[../src/stream.d:14062] [OS-ERROR]: OS-ERROR(98): Address already in use
ERROR!! ERROR should be #\a !
(LISTP (SHOW (LIST (MULTIPLE-VALUE-LIST (SOCKET-STREAM-LOCAL *SOCKET-1*)) (MULTIPLE-VALUE-LIST (SOCKET-STREAM-PEER *SOCKET-1*)) (MULTIPLE-VALUE-LIST (SOCKET-STREAM-LOCAL *SOCKET-2*)) (MULTIPLE-VALUE-LIST (SOCKET-STREAM-PEER *SOCKET-2*))) :PRETTY T))
[SIMPLE-TYPE-ERROR]: SOCKET-STREAM-LOCAL: argument #1=#<CLOSED INPUT BUFFERED SOCKET-STREAM CHARACTER localhost:46709> is not an open SOCKET-STREAM

ERROR!! ERROR should be T !
(SEARCH " (" (SOCKET-STREAM-LOCAL *SOCKET-1* T))
[SIMPLE-TYPE-ERROR]: SOCKET-STREAM-LOCAL: argument #1=#<CLOSED INPUT BUFFERED SOCKET-STREAM CHARACTER localhost:46709> is not an open SOCKET-STREAM

ERROR!! ERROR should be NIL !
(SEARCH " (" (SOCKET-STREAM-PEER *SOCKET-1* T))
[SIMPLE-TYPE-ERROR]: SOCKET-STREAM-PEER: argument #1=#<CLOSED INPUT BUFFERED SOCKET-STREAM CHARACTER localhost:46709> is not an open SOCKET-STREAM

ERROR!! ERROR should be NIL !
(SOCKET-STATUS (CONS *SOCKET-2* :INPUT) 0)
[SIMPLE-TYPE-ERROR]: SOCKET-STATUS: argument #1=#<CLOSED IO INPUT-BUFFERED SOCKET-STREAM CHARACTER 0.0.0.0:46709> is not an open stream

Makefile:25: recipe for target 'tests' failed
make[1]: *** [tests] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/don/hg/clisp/build-dir/tests'
Makefile:2167: recipe for target 'check-tests' failed
make: *** [check-tests] Error 2
[don@localhost build-dir]$ 


same thing happens if I make check as root
netstat --inet -l -n -p
doesn't show any 9090 in use...

Aha - we seem to be using inet6 by default now!
netstat  -l -n -p
shows:
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      2999/sshd
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd
 ...

Re: libffcall and selinux

[<don...@is...>]

I turned off the program using port 9090 and retried make check.
It now ends with:
(STRINGP (WITH-OUTPUT-TO-STRING (S) (DESCRIBE (MAKE-ARRAY NIL :ELEMENT-TYPE NIL) S)))
EQL-OK: T
(STRINGP (WITH-OUTPUT-TO-STRING (S) (DESCRIBE NIL S)))
;; connecting to "http://clisp.org/impnotes/id-href.map"...connected...HTTP/1.1 200 OK...74,909 bytes
;; SYSTEM::GET-STRING-MAP(#<IO INPUT-BUFFERED SOCKET-STREAM CHARACTER clisp.org:80>)...Makefile:25: recipe for target 'tests' failed
make[1]: *** [tests] Segmentation fault (core dumped)
make[1]: Leaving directory '/home/don/hg/clisp/build-dir/tests'
Makefile:2167: recipe for target 'check-tests' failed
make: *** [check-tests] Error 2

system log still contains no AVC, but does contain:
May 25 12:09:22 localhost audit: ANOM_ABEND auid=0 uid=0 gid=0 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=10960 comm="systemd-tty-ask" exe="/usr/bin/systemd-tty-ask-password-agent" sig=11
May 25 12:10:41 localhost audit: ANOM_ABEND auid=0 uid=0 gid=0 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=11356 comm="lisp.run" exe="/home/don/hg/clisp/build-dir/lisp.run" sig=11
May 25 12:10:41 localhost kernel: lisp.run[11356]: segfault at b000000b1f1 ip 00000000004bdeb4 sp 00007ffd4d6b82f0 error 4 in lisp.run[400000+2c1000]
May 25 12:10:42 localhost abrt-hook-ccpp: Process 11356 (lisp.run) of user 0 killed by SIGSEGV - dumping core

I'm guessing this is related to the problem you're now working on,
but not related to selinux.

make -k check also shows
 > form: (nth-prime 150)

*** - Program stack overflow. RESET


Let me know if I can do anything else useful with this VM before 
I turn it off.

Re: libffcall and selinux

[Bruno H. <br...@cl...>]

Hi Don,

> May 25 12:10:41 localhost audit: ANOM_ABEND auid=0 uid=0 gid=0 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 pid=11356 comm="lisp.run" exe="/home/don/hg/clisp/build-dir/lisp.run" sig=11
> May 25 12:10:41 localhost kernel: lisp.run[11356]: segfault at b000000b1f1 ip 00000000004bdeb4 sp 00007ffd4d6b82f0 error 4 in lisp.run[400000+2c1000]
> May 25 12:10:42 localhost abrt-hook-ccpp: Process 11356 (lisp.run) of user 0 killed by SIGSEGV - dumping core
> 
> I'm guessing this is related to the problem you're now working on,
> but not related to selinux.

Yes, nearly everyone on Linux/x86_64 is seeing this (or a similar) crash.

> make -k check also shows
>  > form: (nth-prime 150)
> 
> *** - Program stack overflow. RESET

Now this means that the default stack size is quite small. What's the result of
$ ulimit -a | grep stack
? For me, it's:
stack size              (kbytes, -s) 8192

> Let me know if I can do anything else useful with this VM before 
> I turn it off.

Yes, it would be useful to enable SELinux checks (at least those that smell
like related to "restricted mprotect" or like "write xor execute" or similar,
and see how clisp behaves with these checks enabled.

Bruno

Re: libffcall and selinux

[<don...@is...>]

Bruno Haible writes:
 > > make -k check also shows
 > >  > form: (nth-prime 150)
 > > 
 > > *** - Program stack overflow. RESET
 > 
 > Now this means that the default stack size is quite small. What's
 > the result of
 > $ ulimit -a | grep stack
 > ? For me, it's:
 > stack size              (kbytes, -s) 8192

stack size              (kbytes, -s) 8515

I was surprised not to see anything about setting limit
at the end of the configure output.

 > > Let me know if I can do anything else useful with this VM before 
 > > I turn it off.
 > 
 > Yes, it would be useful to enable SELinux checks (at least those
 > that smell like related to "restricted mprotect" or like "write xor
 > execute" or similar, and see how clisp behaves with these checks
 > enabled.

So far I've left the default selinux config alone.
Let me know how to determine whether it contains the checks you want,
and if not, how to add them, or if you don't happen to know, any advice
you can offer on how to find out.

Also by "how clisp behaves" you mean what part of the process?
Just make check?  Or start over from building libsigsegv and ffcall?

Re: libffcall and selinux

[Bruno H. <br...@cl...>]

Don,

> Let me know how to determine whether it contains the checks you want,
> and if not, how to add them, or if you don't happen to know, any advice
> you can offer on how to find out.

I don't know. Maybe the docs about SELinux, from Red Hat, have some infos?

> Also by "how clisp behaves" you mean what part of the process?
> Just make check?  Or start over from building libsigsegv and ffcall?

Start with "make check". If that fails miserably, then reconfigure,
rebuild, and reinstall libffcall, then rebuild clisp (here, I'd guess,
reconfigure is not necessary, just "make clean && make").

Bruno