clisp-devel — closed list for developers and testers

Re: build-porting64-gcc-debug_gcsafety

[Bruno H. <br...@cl...>]

Hi Sam,

> $ make -f Makefile.devel multibuild-porting64-gcc
> ...
> ../src/lispbibl.d:323:83: fatal error: intparam.h: No such file or directory
>    #include "intparam.h"  /* integer-type characteristics created by the machine */

I've now added a message to configure that, in this case, tell you to look in
config.log.

Errors at this stage frequently come from unsuitable values in the environment
variables CC, CXX, and MULTIBUILD_64_OPTIONS.

Bruno

build-porting64-gcc-debug_gcsafety

[Sam S. <sd...@gn...>]

Hi Bruno,
This is my first attempt to use your new multibuild targets.
I did this:

--8<---------------cut here---------------start------------->8---
$ make -f Makefile.devel multibuild-porting64-gcc
...
./comment5 ../src/aridecl.d | ./gctrigger | ./varbrace > aridecl.c
End of comment outside comment in line 154
End of comment outside comment in line 199
g++ -I/home/sds/src/clisp/current/src -I/home/sds/src/clisp/current/build-porting64-gcc-debug_gcsafety/gllib -I/home/sds/src/clisp/current/src/gllib -g -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wno-sign-compare -Wno-format-nonliteral -Wno-invalid-offsetof -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DDEBUG_GCSAFETY -DENABLE_UNICODE -DDYNAMIC_FFI -c spvw.c
In file included from ../src/spvw.d:23:0:
../src/lispbibl.d:323:83: fatal error: intparam.h: No such file or directory
   #include "intparam.h"  /* integer-type characteristics created by the machine */
                                                                                   ^
compilation terminated.
Makefile:1066: recipe for target 'spvw.o' failed
make[1]: *** [spvw.o] Error 1
make[1]: Leaving directory '/home/sds/src/clisp/current/build-porting64-gcc-debug_gcsafety'
--8<---------------cut here---------------end--------------->8---

This is a vanilla ubuntu 16.10 (Linux 4.8.0-40-generic x86_64
gcc (Ubuntu 6.2.0-5ubuntu12) 6.2.0 20161005).



-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.10 (yakkety) X 11.0.11804000
http://www.childpsy.net/ http://palestinefacts.org
http://memri.org http://www.memritv.org http://ffii.org
Any connection between your reality and mine is purely coincidental.

Re: Modify the bug#679 fix

[Sam S. <sd...@gn...>]

> * Bruno Haible <oe...@py...t> [2017-03-09 01:56:50 +0100]:
>
> #9  0x0819dc5b in ngci_pointable (obj=...) at ../src/lispbibl.d:7260
> #10 0x081a035f in file_stream_truename (stream=...) at ../src/pathname.d:889

fixed, thanks

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il
http://islamexposedonline.com http://americancensorship.org http://no2bds.org
It is possible to be both rich and honest. First you become rich, then
honest.

Re: Modify the bug#679 fix

[Bruno H. <br...@cl...>]

Hi Sam,

Thanks for your fix, along the discussed lines. However, there is a GC-safety bug.

How to reproduce:
- Set MULTIBUILD_32_OPTIONS so that the build will find libsigsegv and optionally
  also readline and libffcall.
- $ CC="gcc -m32" CXX="g++ -m32" make -f Makefile.devel build-porting32-gcc-debug_gcsafety

It crashes like this:

./lisp.run -B . -N locale -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc -q -M lispinit.mem -x '(truename (make-stream :output))'
STACK size: 98206 [0xf6fc3f00 0xf6f64088]
Exiting on signal 6

When I look at it in gdb:
$ gdb lisp.run core
(gdb) where
...
#8  0xf73e6207 in abort () from /lib32/libc.so.6
#9  0x0819dc5b in ngci_pointable (obj=...) at ../src/lispbibl.d:7260
#10 0x081a035f in file_stream_truename (stream=...) at ../src/pathname.d:889
#11 0x081b9e3b in C_truename () at ../src/pathname.d:5630
...

Bruno

Re: asciz_equal & asciz_length & strncmp

[Bruno H. <br...@cl...>]

Hi Sam,

> What's the status of these functions?

asciz_equal and asciz_length are the usual functions to use on C strings.

> Are they officially obsolete?

No.

> ISTR that strlen and strmp were avoided because they used registers that
> CLISP reserved for itself - it this still the case?

I avoided strlen and strcmp because
  * As you say, we cannot expect that libc functions respect our choices
    of reserved CPU registers. Thus the need to wrap calls of strlen and
    strcmp between begin_system_call() / end_system_call().
  * I also like the naming because "asciz" is just one data type among
    many others, and we have a function named asciz_to_string. In other
    words, the C and POSIX standards have terrible naming conventions.
    (Could you tell me, without looking it up, what is the difference
    between memcpy, mempcpy, and memccpy?)

> When can I use strncmp(3)? (we don't have a replacement)

strncmp is very easily used in the wrong way. (I guess I do it wrong
1 out of 2 times.) I prefer APIs that are maybe less general, but have
a semantics that is easier to grasp, such as
  boolean asciz_startswith (const char* s, const char* prefix)
or
  /* If s starts with the prefix, return the portion of s after the prefix,
     otherwise return NULL. */
  const char* asciz_tail (const char* s, const char* prefix)

Bruno

Re: ‘rl_readline_state_t’ undeclared

[Bruno H. <br...@cl...>]

Hi Sam,

> > libsvm.c: In function ‘module__libsvm__init_function_1’:
> > libsvm.c:44:57: error: ‘rl_readline_state_t’ undeclared (first use in 
> > this function)
> >  
> > register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);
> 
> Yes, this is an easy to reproduce bug.
> 
> The reason is that `(def-c-type foo)` (without definition) modifies
> *c-type-table* and it becomes a part of CLISP base image (because
> readline is a base module).
> Now, when writing the C file after compiling FFI forms, CLISP writes
> register_foreign_inttype for all such types.

When 'clisp-link' is used to create a linking set B from a linking set A
and some modules M1, M2, ... [in our case A = base, B = full]: why does
it need to do anything with the types that are already in *c-type-table*
of A ?

Is clisp-link starting out with the 'boot' memory image and adding the
modules from A and the modules M1, M2, ... to it?

Or is clisp-link starting out with the memory image from A and adding the
modules M1, M2, ... to it?

If it's the first case, then why? Why not use the second approach?

If it's the second case, then why the need to rebuild the *c-ctype-table* from
scratch?

(Sorry, I didn't have time to look how the code does it.)

Bruno

asciz_equal & asciz_length & strncmp

[Sam S. <sd...@gn...>]

What's the status of these functions?
Are they officially obsolete?
ISTR that strlen and strmp were avoided because they used registers that
CLISP reserved for itself - it this still the case?
When can I use strncmp(3)? (we don't have a replacement)

-- 
Sam Steingold (http://sds.podval.org/) on Ubuntu 16.10 (yakkety) X 11.0.11804000
http://www.childpsy.net/ http://iris.org.il http://memri.org
http://islamexposedonline.com http://camera.org http://think-israel.org
Two magic words will open almost all doors for you: "pull" and "push".

Re: ‘rl_readline_state_t’ undeclared

[Sam S. <sd...@gn...>]

Hi Ken,

> * Ken Brown <xoebja@pbearyy.rqh> [2017-03-06 13:38:58 -0500]:
>
> You introduced rl_readline_state_t in the following commit:
>
> changeset:   15750:06790b3788cf
> user:        Bruno Haible <br...@cl...>
> date:        Sun Feb 26 12:07:24 2017 +0100
> files:       modules/readline/config.h.in modules/readline/configure 
> modules/readline/configure.in modules/readline/readline.lisp src/configure
> description:
> readline: Update binding to avoid build failure with GNU readline 7.0.
>
> I'm now getting errors like the following for several of the modules I 
> build:
>
> gcc 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/gllib 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src/gllib 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/gllib 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src/gllib 
>   -ggdb -O2 -pipe -Wimplicit-function-declaration -W -Wswitch -Wcomment 
> -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit 
> -Wno-sign-compare -Wno-format-nonliteral -O2 -fexpensive-optimizations 
> -fno-strict-aliasing -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES 
> -DDLL_EXPORT -DPIC 
> -I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/linkkit 
> -c libsvm.c
> libsvm.c:18:6: warning: no previous declaration for ‘svm_destroy_model’ 
> [-Wmissing-declarations]
>   void svm_destroy_model (struct svm_model *model){ 
> svm_free_and_destroy_model(&model); }
>        ^
> libsvm.c: In function ‘module__libsvm__init_function_1’:
> libsvm.c:44:57: error: ‘rl_readline_state_t’ undeclared (first use in 
> this function)
>  
> register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);

Yes, this is an easy to reproduce bug.

The reason is that `(def-c-type foo)` (without definition) modifies
*c-type-table* and it becomes a part of CLISP base image (because
readline is a base module).
Now, when writing the C file after compiling FFI forms, CLISP writes
register_foreign_inttype for all such types.

The solution is probably to write such foreign inttypes into a
file-specific table rather than the global one.

Thanks for reporting the bug!

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://www.dhimmitude.org
http://jij.org http://iris.org.il http://camera.org http://no2bds.org
Bug free software merely has random features.

AW: next nightly build problem

[<Joe...@t-...>]

Hi,

Don Cohen wrote:

[3]> (software-type)
"gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DDYNAMIC_MODULES  libgnu.a  -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  /usr/local/lib64/libsigsegv.a -lc
SAFETY=3 TYPECODES WIDE_HARD SPVW_BLOCKS SPVW_PURE SINGLEMAP_MEMORY libsigsegv 2.10 libffcall 1.13"

Good point! Here's my MS-VC version (32 bit) for comparison.
[13]> (software-type)
"cl  -G5 -Ot -Oy -Ob1 -Gs -Gf -Gy -Og -W4 -DUNICODE -DDYNAMIC_FFI -DNO_GETTEXT -I.  charset.lib avcall.lib callback.lib  user32.lib ws2_32.lib advapi32.lib ole32.lib shell32.lib sigsegv.lib 
SAFETY=1 HEAPCODES STANDARD_HEAPCODES GENERATIONAL_GC SPVW_BLOCKS SPVW_MIXED TRIVIALMAP_MEMORY
libsigsegv 2.2"

As you can see, all memory options are almost opposite.

Regards,
	Jörg

Re: next nightly build problem

[<Joe...@t-...>]

Hi,

>> >I don't even know how to check in lisp what memory model is in use.
>> It'd be guesswork.
>Call (software-version).

My rather ancient CLISP for MS-Windows replies:
[12]> (software-version)
"C compiler" ; this ain't tip
[13]>

:-)
	Jörg

Re: next nightly build problem

[<don...@is...>]

Bruno Haible writes:
 > > Don Cohen wrote:
 > > >I don't even know how to check in lisp what memory model is in use.
 > > It'd be guesswork.
 > 
 > Call (software-version).

not quite, but turned out to be a good hint:

[1]> (software-version)
"GNU C 6.3.1 20161221 (Red Hat 6.3.1-1)"
[2]> (apropos "SOFTWARE")
SOFTWARE-TYPE                              function
SOFTWARE-VERSION                           function

[3]> (software-type)
"gcc -m64 -g -O2 -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -fno-strict-aliasing -ggdb -O0 -DDEBUG_OS_ERROR -DDEBUG_SPVW -DDEBUG_BYTECODE -DSAFETY=3 -DENABLE_UNICODE -DNO_TERMCAP_NCURSES -DDYNAMIC_FFI -DDYNAMIC_MODULES  libgnu.a  -ldl /usr/local/lib/libavcall.a /usr/local/lib/libcallback.a  /usr/local/lib64/libsigsegv.a -lc  
SAFETY=3 TYPECODES WIDE_HARD SPVW_BLOCKS SPVW_PURE SINGLEMAP_MEMORY
libsigsegv 2.10
libffcall 1.13"
[4]> 

Re: next nightly build problem

[Bruno H. <br...@cl...>]

> Don Cohen wrote:
> >I don't even know how to check in lisp what memory model is in use.
> It'd be guesswork.

Call (software-version).

Bruno

Re: ‘rl_readline_state_t’ undeclared

[Ken B. <kb...@co...>]

On 3/6/2017 4:04 PM, Ken Brown wrote:
> It's interesting that the problem occurs with four consecutive modules
> but then is OK again.  I'll try experimenting with various
> rearrangements of the list of modules.

That was a red herring.  I get the problem with the same four modules, 
regardless of order.

Ken

Re: ‘rl_readline_state_t’ undeclared

[Ken B. <kb...@co...>]

On 3/6/2017 3:55 PM, Ken Brown wrote:
> Hi Bruno,
>
> On 3/6/2017 2:49 PM, Bruno Haible wrote:
>>> register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);
>>
>> How come that the initialization function of the libsvm module references
>> types from another, unrelated module??
>
> No idea.  The modules I normally build on 32-bit Cygwin [other than
> clx/new-clx, which is currently broken as you know] are rawsock, dirkey,
> berkeley-db, pcre, gdbm, bindings/win32, postgresql, fastcgi, zlib,
> libsvm, gtk2, and dbus.  The problem occurs with fastcgi, zlib, libsvm,
> and gtk2, but not with any of the others.
>
>> How do you reproduce it?
>
> ./configure --with-modules=rawsock --with-modules=dirkey
> --with-modules=berkeley-db --with-modules=pcre --with-modules=gdbm
> --with-modules=bindings/win32 --with-modules=postgresql
> --with-modules=fastcgi --with-modules=zlib --with-modules=libsvm
> --with-modules=gtk2 --with-modules=dbus build_dir
>
> cd build_dir
>
> make -k MODULES="rawsock dirkey berkeley-db pcre gdbm bindings/win32
> postgresql fastcgi zlib libsvm gtk2"

Typo: It should have been "...gtk2 dbus".

It's interesting that the problem occurs with four consecutive modules 
but then is OK again.  I'll try experimenting with various 
rearrangements of the list of modules.

Ken

Re: ‘rl_readline_state_t’ undeclared

[Ken B. <kb...@co...>]

Hi Bruno,

On 3/6/2017 2:49 PM, Bruno Haible wrote:
>> register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);
>
> How come that the initialization function of the libsvm module references
> types from another, unrelated module??

No idea.  The modules I normally build on 32-bit Cygwin [other than 
clx/new-clx, which is currently broken as you know] are rawsock, dirkey, 
berkeley-db, pcre, gdbm, bindings/win32, postgresql, fastcgi, zlib, 
libsvm, gtk2, and dbus.  The problem occurs with fastcgi, zlib, libsvm, 
and gtk2, but not with any of the others.

> How do you reproduce it?

./configure --with-modules=rawsock --with-modules=dirkey 
--with-modules=berkeley-db --with-modules=pcre --with-modules=gdbm 
--with-modules=bindings/win32 --with-modules=postgresql 
--with-modules=fastcgi --with-modules=zlib --with-modules=libsvm 
--with-modules=gtk2 --with-modules=dbus build_dir

cd build_dir

make -k MODULES="rawsock dirkey berkeley-db pcre gdbm bindings/win32 
postgresql fastcgi zlib libsvm gtk2"

Ken

Re: ‘rl_readline_state_t’ undeclared

[Bruno H. <br...@cl...>]

Hi Ken,

> libsvm.c: In function ‘module__libsvm__init_function_1’:
> libsvm.c:44:57: error: ‘rl_readline_state_t’ undeclared (first use in 
> this function)
>  
> register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);

How come that the initialization function of the libsvm module references
types from another, unrelated module??

How do you reproduce it?

> shouldn't the 
> typedef of rl_readline_state_t have been propagated to some header used 
> in building other modules that rely on readline?

AFAICS, libsvm and readline are independent.

Bruno

‘rl_readline_state_t’ undeclared

[Ken B. <kb...@co...>]

Hi Bruno,

You introduced rl_readline_state_t in the following commit:

changeset:   15750:06790b3788cf
user:        Bruno Haible <br...@cl...>
date:        Sun Feb 26 12:07:24 2017 +0100
files:       modules/readline/config.h.in modules/readline/configure 
modules/readline/configure.in modules/readline/readline.lisp src/configure
description:
readline: Update binding to avoid build failure with GNU readline 7.0.

I'm now getting errors like the following for several of the modules I 
build:

gcc 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/gllib 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src/gllib 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/gllib 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/src/clisp/src/gllib 
  -ggdb -O2 -pipe -Wimplicit-function-declaration -W -Wswitch -Wcomment 
-Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit 
-Wno-sign-compare -Wno-format-nonliteral -O2 -fexpensive-optimizations 
-fno-strict-aliasing -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES 
-DDLL_EXPORT -DPIC 
-I/home/kbrown/src/cygclisp/clisp-2.49-1.20170303hg15769.i686/build/linkkit 
-c libsvm.c
libsvm.c:18:6: warning: no previous declaration for ‘svm_destroy_model’ 
[-Wmissing-declarations]
  void svm_destroy_model (struct svm_model *model){ 
svm_free_and_destroy_model(&model); }
       ^
libsvm.c: In function ‘module__libsvm__init_function_1’:
libsvm.c:44:57: error: ‘rl_readline_state_t’ undeclared (first use in 
this function)
 
register_foreign_inttype("rl_readline_state_t",sizeof(rl_readline_state_t),(rl_readline_state_t)-1<=(rl_readline_state_t)0);


This is on Cygwin.  I don't currently have access to a Linux system on 
which I can build clisp, so I don't know if the problem is Cygwin-specific.

My knowledge of how modules work is very limited, but shouldn't the 
typedef of rl_readline_state_t have been propagated to some header used 
in building other modules that rely on readline?

Ken

Re: next nightly build problem

[<don...@is...>]

Joe...@t-... writes:

 > The memory models influence some limits, e.g. MOST-POSITIVE-FIXNUM and
 > perhaps ARRAY-TOTAL-SIZE-LIMIT or one of the siblings.
 > Weren't you the one once wondering about the seemingly low
 > string size limit despite a 64 bit build?

Yes, in fact I found myself wondering about this again just yesterday.
And now I can't seem to find the answer.  
What is the reason for this, and where should I have looked for it?

Re: next nightly build problem

[<Joe...@t-...>]

Hi,

Don Cohen wrote:
>I don't even know how to check in lisp what memory model is in use.
It'd be guesswork.

Without libsigsegv or generational GC:
(time (ext:gc)) will be slow (seconds on very old machines).
GC in the milliseconds range is a feature of generational GC.

The memory models influence some limits, e.g. MOST-POSITIVE-FIXNUM and
perhaps ARRAY-TOTAL-SIZE-LIMIT or one of the siblings.
Weren't you the one once wondering about the seemingly low
string size limit despite a 64 bit build?

Regards,
	Jörg

Re: streams & truename

[Bruno H. <br...@cl...>]

Hi Sam,

> We have a problem, yet another case where the CL universe is different
> from UNIX (and, probably, Windows, but I don't have access to that, so,
> thankfully, I can excuse myself):
> 
> * CL assumes that (truename open-file-stream) works

Yes and no.

CL says that it is OK to call (truename open-file-stream).
[See http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/fun_truename.html
 The argument is a 'pathname designator'. The glossary says that this
 includes 'stream associated with a file', and 'stream associated with a file'
 includes 'file stream'.]

CL also says that the call may fail.
[Exceptional Situations:
 An error of type file-error is signaled if an appropriate file cannot be located
 within the file system for the given filespec, or if the file system cannot
 perform the requested operation.]

Cases where TRUENAME fails, on Unix, are when you have read and search access to the
current directory but not to one of the ancestor directories.

> * on unix, "/dev/fd/0" resolves to "/proc/<pid>/fd/0" which is a symlink
>   to "pipe:[123456789]" when the process is run like "clisp | cat".
>   (and the "/proc/<pid>/fd/pipe:[...]" file does not exist).
>   IOW, on unix one can do i/o on a non-existent path (sometimes).

Yes, I can reproduce this:
$ ./clisp -norc -q -x '(truename "/dev/fd/1")'
#P"/dev/pts/9"
$ ./clisp -norc -q -x '(truename "/dev/fd/1")' | cat
*** - TRUENAME: File #P"/proc/872/fd/pipe:[3163863]" does not exist

> What to do about it?
> 
> *1* We can detect that "/proc/<pid>/fd/0" is on "/proc" and refuse to
>   follow symlinks there.
>   This is fairly arbitrary, but we already use this sort of magic when
>   deciding the default value for the :buffered argument.
>   This would also prevent users from detecting that the stdout is a
>   path (not good).
>   [btw, shouldn't we treat /proc and /dev identically? neither is a real
>   file system].
> 
> *2* We can defer computing strm_file_truename until necessary, leading to
>   various special cases for (open (make-stream :input)) (which will call
>   truename) &c.
> 
> I don't particularly like either approach; but I think I dislike the
> first one less.

You need both.

You need *1* because the two commands I gave above show that the problem
already occurs when streams are not involved.

You need *2* because no one has asked to compute the truename ahead of time,
when the stream is being created.

More details about *1*:

When the symlink chain is /dev/fd/1 -> /proc/self/fd/1 -> /dev/pts/9
I want to get "/dev/pts/9" as result. Whereas when the symlink chain is
/dev/fd/1 -> /proc/self/fd/1 -> pipe:[3163863]  or
/dev/fd/1 -> /proc/self/fd/1 -> socket:[2881830]
I want to get "/proc/self/fd/1" as result, because you can't do anything with
the internal inode number of the pipe or socket.
Therefore it's only some of the symlinks in /proc that you need to refuse to
follow.

/dev is, like autofs, binfmt_misc, cgroup, debugfs, devtmpfs, efivarfs,
fusectl, hugetlbfs, mqueue, proc, pstore, securityfs, sysfs, tmpfs, tracefs
(and sure some others) a synthetic file system, that is, a file system not
backed by a disk. But its symlinks are OK, as far as I have seen. So I would
leave /dev alone until we see that it has problems.

More details about *2*:

Following these links in the hyperspec
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/fun_truename.html
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/glo_p.html#pathname_designator
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/glo_s.html#stream_associated_with_a_file
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/glo_f.html#file_stream
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/syscla_file-stream.html
http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/fun_open.html
I cannot see a wording that indicates that the truename needs to be stored
in the stream. Rather it looks like the stream is supposed to store the
(not canonicalized) pathname to which it was associated when it was created,
and the TRUENAME function will canonicalize it when TRUENAME is called.

It it harmful to call TRUENAME ahead of time because the user has not asked for it
when he creates a stream, and:
  - TRUENAME may fail (see above),
  - File systems nowadays are optimized so that very few operations need to
    canonicalize a file name. This canonicalization is expensive, even when
    done through a kernel API [1].

In particular

$ ./clisp -norc -q -x '(open "/dev/fd/1")' | cat
*** - OPEN: File #P"/proc/1040/fd/pipe:[3172140]" does not exist

is wrong. The correct output would be

$ ./clisp -norc -q -x '(open "/dev/fd/1")' | cat
#<INPUT UNBUFFERED FILE-STREAM CHARACTER #P"/dev/fd/1" @1>

Bruno

[1] http://stackoverflow.com/questions/1188757/getting-filename-from-file-descriptor-in-c

Re: next nightly build problem

[<don...@is...>]

Bruno Haible writes:
 > Hi Don,
 > 
 > >  > - If you are interested in a build that works, choose one of those with a
 > >  >   large cbcstep3.log.
 > > 
 > > I'm hoping before long they'll all work.
 > 
 > It's better now: on Linux/x86_64 all pass for me, except for

$ ls -l build-*/cbcstep3.log
-rw-rw-r--. 1 don don 3043905 Mar  5 14:33 build-porting64-gcc-generational_gc-multithread_gc/cbcstep3.log
-rw-rw-r--. 1 don don 3040454 Mar  5 14:31 build-porting64-gcc-generational_gc-old_gc/cbcstep3.log
-rw-rw-r--. 1 don don 3039228 Mar  5 14:09 build-porting64-gcc-generic64_heapcodes/cbcstep3.log
-rw-rw-r--. 1 don don 3034785 Mar  5 14:03 build-porting64-gcc-heapcodes/cbcstep3.log
-rw-rw-r--. 1 don don 3043202 Mar  5 14:18 build-porting64-gcc-heapcodes-spvw_mixed_blocks/cbcstep3.log
-rw-rw-r--. 1 don don 3040171 Mar  5 14:12 build-porting64-gcc-heapcodes-spvw_pages/cbcstep3.log
-rw-rw-r--. 1 don don 3036625 Mar  5 14:28 build-porting64-gcc-multithread_gc/cbcstep3.log
-rw-rw-r--. 1 don don 3033154 Mar  5 14:26 build-porting64-gcc-old_gc/cbcstep3.log
-rw-rw-r--. 1 don don 3035492 Mar  5 13:58 build-porting64-gcc-portability/cbcstep3.log
-rw-rw-r--. 1 don don 3033094 Mar  5 14:36 build-porting64-gcc-safety0/cbcstep3.log
-rw-rw-r--. 1 don don 3037978 Mar  5 14:38 build-porting64-gcc-safety0-optimized/cbcstep3.log
-rw-rw-r--. 1 don don 3033187 Mar  5 14:01 build-porting64-gcc-safety3/cbcstep3.log
-rw-rw-r--. 1 don don 3037636 Mar  5 14:23 build-porting64-gcc-spvw_pure_blocks/cbcstep3.log
-rw-rw-r--. 1 don don 3038762 Mar  5 14:06 build-porting64-gcc-standard_heapcodes/cbcstep3.log
-rw-rw-r--. 1 don don 3042771 Mar  5 14:20 build-porting64-gcc-typecodes-spvw_mixed_blocks/cbcstep3.log
-rw-rw-r--. 1 don don 3039694 Mar  5 14:15 build-porting64-gcc-typecodes-spvw_pages/cbcstep3.log

 > build-porting64-gcc-spvw_pure_blocks, which triggers a Heisenbug:

Does that mean it sometimes appears and sometimes does not?

 >   *** - APPLY: too few arguments for #<STANDARD-GENERIC-FUNCTION COMPUTE-SLOTS>
 > I don't know how to track down this one at the moment.

I don't see that in my output.


 > Yes, you guessed right. I would also revert such a local change before
 > doing "hg pull --update" and then re-apply it.

Currently my nightly build script does a pull.
It wouldn't make much sense to patch and unpatch as part of
that script.

It sounds like for now I should leave the patch in place 
so I've done (after generating the output above)
  hg diff -c d61ff18e7daa | patch -p1

and when that gets fixed so the nightly builds work again,
I'll try adding multibuild port to the script.
It looks like I have to first do
 rm -rf build-*
or make thinks there's nothing to do.

Do you think it's adequate to just look for logs that are smaller than
3 MB as an indicator of failure, or look at tails of the logs as I now
do or something else?

BTW, a sample of what I see there:

  $ tail build-porting64-gcc-standard_heapcodes/cbcstep3.log

  WARNING: SYSTEM::ENSURE-IMPNOTES-MAP: invalid symbol "POSIX:CLIPBOARD" with id
           "clipboard":
           READ from #<INPUT STRING-INPUT-STREAM>: #<PACKAGE POSIX> has no
           external symbol with name "CLIPBOARD"

  662 IDs
  "http://clisp.org/beta/impnotes/"
  Bye.
  make[1]: Leaving directory '/home/don/hg/clisp/build-porting64-gcc-standard_heapcodes'

My normal nightly test creates both a mt and a non-mt clisp and 
then an ap5 image in each.  Since there is some #+/- mt in the
current ap5 code, I do want to test each of those, but I don't
think I have anything that depends on memory models.
I don't even know how to check in lisp what memory model is in use.

I see two appearances of multithread in the multi build port list.
Are those independent of #+MT ?  Using a multi thread gc which is
independent of #+MT ?  Are any or all of the tests above #+MT ?

Re: streams & truename

[Sam S. <sd...@gn...>]

Bruno,

We have a problem, yet another case where the CL universe is different
from UNIX (and, probably, Windows, but I don't have access to that, so,
thankfully, I can excuse myself):

* CL assumes that (truename open-file-stream) works

* on unix, "/dev/fd/0" resolves to "/proc/<pid>/fd/0" which is a symlink
  to "pipe:[123456789]" when the process is run like "clisp | cat".
  (and the "/proc/<pid>/fd/pipe:[...]" file does not exist).
  IOW, on unix one can do i/o on a non-existent path (sometimes).

What to do about it?

* We can detect that "/proc/<pid>/fd/0" is on "/proc" and refuse to
  follow symlinks there.
  This is fairly arbitrary, but we already use this sort of magic when
  deciding the default value for the :buffered argument.
  This would also prevent users from detecting that the stdout is a
  path (not good).
  [btw, shouldn't we treat /proc and /dev identically? neither is a real
  file system].

* We can defer computing strm_file_truename until necessary, leading to
  various special cases for (open (make-stream :input)) (which will call
  truename) &c.

I don't particularly like either approach; but I think I dislike the
first one less.

Other ideas?

> * Bruno Haible <oe...@py...t> [2017-02-27 22:25:59 +0100]:
>
> $ ./clisp -K boot  -E UTF-8 -Emisc 1:1 -Epathname 1:1 -norc --version 2>/dev/null | \
>   hexdump -e '"%06.6_ax  " 16/1 "%02X "' -e '"  " 16/1 "%_p" "\n"' "$@"
> 000000  0A 2A 2A 2A 20 2D 20 54 52 55 45 4E 41 4D 45 3A  .*** - TRUENAME:
> 000010  20 54 68 65 20 76 61 6C 75 65 20 6F 66 20 2A 54   The value of *T
> 000020  45 52 4D 49 4E 41 4C 2D 49 4F 2A 20 77 61 73 20  ERMINAL-IO* was 
> 000030  6E 6F 74 20 61 6E 20 61 70 70 72 6F 70 72 69 61  not an appropria
> 000040  74 65 20 73 74 72 65 61 6D 3A 20 23 3C 43 4C 4F  te stream: #<CLO
> 000050  53 45 44 20 49 4F 20 54 45 52 4D 49 4E 41 4C 2D  SED IO TERMINAL-
> 000060  53 54 52 45 41 4D 3E 2E 20 49 74 20 68 61 73 20  STREAM>. It has 
> 000070  62 65 65 6E 20 63 68 61 6E 67 65 64 20 74 6F 0A  been changed to.
> 000080  20 20 20 20 20 20 23 3C 49 4F 20 54 45 52 4D 49        #<IO TERMI
> 000090  4E 41 4C 2D 53 54 52 45 41 4D 3E 2E 0A 45 78 69  NAL-STREAM>..Exi
> 0000a0  74 69 6E 67 20 6F 6E 20 73 69 67 6E 61 6C 20 36  ting on signal 6
> 0000b0  0A                                               .
>
> Sam, your turn, I guess.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://www.dhimmitude.org
http://iris.org.il http://no2bds.org http://camera.org https://ffii.org
Life is like Tetris: failures accumulate, successes fade.

Re: Build failure with fastcgi module

[Ken B. <kb...@co...>]

Hi Bruno,

On 3/5/2017 2:53 PM, Bruno Haible wrote:
> Hi Ken,
>
>> I'd like to try removing all uses of '#include <windows.h>' from the Cygwin build.
>> Cygwin is a Posix platform, and trying to mix the Posix API with the Windows API
>> often causes problems.
>
> Do you mean
>   a) to not use Windows API on Cygwin?
>   b) keep using the Windows API on Cygwin, but don't include <windows.h> from
>      lispbibl.d and clisp.h.
>
> a) If you know good replacements for our uses of the Windows API?
>
> b) I would advise against this.
>
>    In a world where the system's include files are all standalone and not
>    conflicting (like glibc systems), it's OK if every compilation unit does
>    only the #includes that it needs.
>
>    But in a world where the include files define types differently, have to
>    be included in a particular order, or define identifiers like ULONGLONG
>    that belong in the programmer's namespace, it's less effort in the long
>    run to use the same includes for all compilation units. This is how
>    lispbibl.d started out, and this is why many other projects also have
>    one main include file.
>
>    With Cygwin and <windows.h> - which we DO need to mix in some places -
>    we are in the second case.

I see your point.  And in any case, removing the use of windows.h is 
much more complicated than I thought.  I spent a few hours on this, and 
every change I made led to a new problem that had to be fixed.  I 
withdraw my suggestion.

Thanks for all your work on clisp.  It's nice to see development resuming.

Ken

Re: Build failure with fastcgi module

[Ken B. <kb...@co...>]

Hi Bruno,

On 3/5/2017 2:40 PM, Bruno Haible wrote:
> But more fundamentally, the use of gnulib is fundamentally broken regarding
> clisp modules:
>   - The clx module should not even see build/gllib/unistd.h.
>   - The syscalls module should not rely on the getloadavg module from the
>     clisp core. This is most evidently seen by a link error on AIX.
> So, you cannot do anythere here until this fundamental mistake is fixed.
> clisp's modules are designed as separately compilable units; therefore they
> must use *independent* autoconf configurations; therefore their uses of
> gnulib must be independent as well.

I wondered about that.  I'm glad to hear that the current behavior is a 
mistake.  So I'll just avoid building clx until this is fixed.

Thanks.

Ken

Re: Build failure with fastcgi module

[Bruno H. <br...@cl...>]

Hi Ken,

> I'd like to try removing all uses of '#include <windows.h>' from the Cygwin build.
> Cygwin is a Posix platform, and trying to mix the Posix API with the Windows API
> often causes problems.

Do you mean
  a) to not use Windows API on Cygwin?
  b) keep using the Windows API on Cygwin, but don't include <windows.h> from
     lispbibl.d and clisp.h.

a) If you know good replacements for our uses of the Windows API?

b) I would advise against this.

   In a world where the system's include files are all standalone and not
   conflicting (like glibc systems), it's OK if every compilation unit does
   only the #includes that it needs.

   But in a world where the include files define types differently, have to
   be included in a particular order, or define identifiers like ULONGLONG
   that belong in the programmer's namespace, it's less effort in the long
   run to use the same includes for all compilation units. This is how
   lispbibl.d started out, and this is why many other projects also have
   one main include file.

   With Cygwin and <windows.h> - which we DO need to mix in some places -
   we are in the second case.

Bruno