#699 CLISP not recognizing spaces in file names

[Jean Louis]

admin-> cat test\ this\ one.lisp 
(print "Hello")

admin-> ./test\ this\ one.lisp 
*** - OPEN: No file name given:
      #P"/home/data1/protected/lib/lisp/quicklisp/dists/quicklisp/software/cl-geos-20160421-git/test/"

basically, if the file has spaces, and I have binfmt_misc system so that lisp files are run by CLISP, there are weird errors appearing.

The same happens if I run the file with spaces directly from command line:

clisp test\ this\ one.lisp 
*** - OPEN: No file name given:
      #P"/home/data1/protected/lib/lisp/quicklisp/dists/quicklisp/software/cl-geos-20160421-git/test/"

admin-> mv test\ this\ one.lisp "one two three.lisp"
[~]
admin-> clisp one\ two\ three.lisp 
*** - LOAD: A file with name one does not exist

Basically CLISP is not recognizing the spaces in file names.

Jean

GNU CLISP 2.49.60+ (2017-06-25) (built 3709299705) (memory 3709303928)
on GNU/Linux variant.

[Jean Louis]

On Thu, Nov 16, 2017 at 06:58:33PM -0000, Sam Steingold wrote:

$ cat foo\ bar.lisp
(format t "~A~%" (lisp-implementation-version))
$ ./clisp -q -norc -K boot foo\ bar.lisp
2.49.60+ (2017-06-25) (built 3719676226) (memory 3719757723)
$ ./clisp -q -norc -K boot "foo bar.lisp"
2.49.60+ (2017-06-25) (built 3719676226) (memory 3719757723)
`` I suspect that yourclisp` is a shell script,
in which case you need to to quote spaces twice

It does work with -q flag, I see. It does not work
without -q flag. I guess it will work with any
flag that allows execution, but not without any
flags before the file. It should though.

file with spaces.lisp:

(princ "Hello")

admin-> clisp -q file\ with\ spaces.lisp

"Hello"

admin-> clisp file\ with\ spaces.lisp
*** - LOAD: A file with name file does not exist

admin-> clisp "file with spaces.lisp"
*** - LOAD: A file with name file does not exist

admin-> clisp -norc "file with spaces.lisp"

"Hello"

In regards that CLISP is shell, this one shows it
is executable:

admin-> file $(realpath /usr/bin/clisp)
/package/prog/clisp/bin/clisp: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, not stripped

Jean

[Sam Steingold]

works for me:

$ cat foo\ bar.lisp
(format t "~A~%" (lisp-implementation-version))
$ ./clisp -q -norc -K boot foo\ bar.lisp
2.49.60+ (2017-06-25) (built 3719676226) (memory 3719757723)
$ ./clisp -q -norc -K boot "foo bar.lisp"
2.49.60+ (2017-06-25) (built 3719676226) (memory 3719757723)

I suspect that your clisp is a shell script, in which case you need to to quote spaces twice.

[Bruno Haible]

This is a consequence of the "Split argv[1] into pieces." hack in lines 180..220 of _clisp.c.
How can the clisp program determine whether it is being invoked via #! or not?
Maybe there are differences in getauxval() that we could exploit??

[Sam Steingold]

yep, and the problem manifests itself only when clisp receives no other arguments.

[Sam Steingold]

https://stackoverflow.com/q/47339792/850781

[Sam Steingold]

https://github.com/Microsoft/WSL/issues/2133 seems to indicate that getauxval(AT_EXECFN) is not Linux-only.
it also appears to be available on gnulib.
looks like Bruno found the solution!

[Bruno Haible]

> https://github.com/Microsoft/WSL/issues/2133 seems to indicate that getauxval(ATEXECFN) is not Linux-only.

No. Linux running in WSL is still a Linux-compatible kernel. No way to make this work from Solaris to FreeBSD.

> it also appears to be available on gnulib.

No. This auxv stuff is among the most unportable things you can ever encounter.

[Jean Louis]

On Fri, Nov 17, 2017 at 02:48:59PM -0000, Bruno Haible wrote:

> https://github.com/Microsoft/WSL/issues/2133 seems to indicate that getauxval(ATEXECFN) is not Linux-only.

No. Linux running in WSL is still a Linux-compatible kernel. No way to make this work from Solaris to FreeBSD.

> it also appears to be available on gnulib.

No. This auxv stuff is among the most unportable things you can ever encounter.

If I may comment:

admin-> perl new\ file.pl

I was using perl much, and perl is pretty
portable, maybe inside there in its code you may
find how they are handling files with spaces.

Jean

[Sam Steingold]

perl follows the standard unix pattern that all single-letter options can be coalesced into one word, so it inherrently does not have to split its argument in scripts.
IOW, clisp does what perl does not have to.

[Bruno Haible]

I think we have the choice between two options:
(a) Use a getauxval based distinction, conditionalized by #if __linux__. This will fix the problems in "normal" use-cases on Linux, and leave it buggy for the other OSes.
(b) Create a second driver program, /usr/bin/clisp!#, next to /usr/bin/clisp, that is compiled from the same _clisp.c with a different conditionalization. So that /usr/bin/clisp can be compiled without the "Split argv[1] into pieces." hack.

(b) solves the problem once and for all, and is therefore my favourite.

[Sam Steingold]

Agreed.
Non-portable linux-only stuff doesn't really solve anything - just makes it harder to figure out the problem from a user complaint.
The /usr/bin/clisp-shebang solution will require modification of all user scripts though...
Note that naming files with #! or !# will lead to a host of issues in makefiles &c.

[Jean Louis]

On Fri, Nov 17, 2017 at 03:06:20PM -0000, Sam Steingold wrote:

Agreed.
Non-portable linux-only stuff doesn't really solve anything - just makes it harder to figure out the problem from a user complaint.
The /usr/bin/clisp-shebang solution will require modification of all user scripts though...
Note that naming files with #! or !# will lead to a host of issues in makefiles &c.
```
admin-> cat test\ this\ one.lisp
(print "Hello")

admin-> ./test\ this\ one.lisp
*** - OPEN: No file name given:
#P"/home/data1/protected/lib/lisp/quicklisp/dists/quicklisp/software/cl-geos-20160421-git/test/"
```

I understand you mention scripts and running it
from a script.

I mentioned it in relation to running a lisp file
from command line. There is no shebang required.

File may contain for example (print "hello") and
if it is run with:

clisp file\ with\ spaces.lisp

then there is error in recognition of the file.

There is no shebang involved there.

Jean

[Sam Steingold]

Jean,
the problem is that when you have a script name with spaces.lisp with #!/usr/bin/clisp-shebang as the first line, the OS executes /usr/bin/clisp-shebang with 1 argument name with spaces.lisp.
When the first line is #!/usr/bin/clisp-shebang -M my-image.mem -E utf-8, the OS executes /usr/bin/clisp-shebang with 2 arguments:

1. -M my-image.mem -E utf-8
2. name with spaces.lisp

Thus the solution is the following:

1. We create and install 2 clisp driver executables:
   clisp-split-1st-arg (named clisp-shebang above) which splits its 1st argument on spaces
   clisp which processes its argv as is
2. The users use clisp at the command line as they do now and in scripts when they do not pass any arguments to it in the first (shebang) line.
3. The users use clisp-split-1st-arg in scripts when they do pass it arguments in the shebang line.

[Jean Louis]

On Sat, Nov 18, 2017 at 01:59:38PM -0000, Bruno Haible wrote:

• status: open --> closed-fixed
• assigned_to: Sam Steingold --> Bruno Haible
• Comment:

> Note that naming files with #! or !# will lead to a host of issues in makefiles &c.

Indeed, Makefile have big trouble with file names or targets that contain '#' characters. But a '!' character works fine. So I called it 'clisp!'. Not a long name, because on Linux the line length is limited to 127 bytes.

> The /usr/bin/clisp-shebang solution will
> require modification of all user scripts
*> though...

That is great.

I have tested it and I found that if I use -norc
flag even if I do not have .clisprc.lisp that
additional new line is printed in this case below,
and I cannot know if that is a bug or not, it
looks suspicious.

admin [ /sources/gnu/clisp-clisp ]$ cat /tmp/file\ with\ spaces.lisp
(print "hello")
admin [ /sources/gnu/clisp-clisp ]$

admin [ /sources/gnu/clisp-clisp/build-portability2 ]$ ./clisp /tmp/file\ with\ spaces.lisp

"hello"
admin [ /sources/gnu/clisp-clisp/build-portability2 ]$

admin [ /sources/gnu/clisp-clisp/build-portability2 ]$ ./clisp -q /tmp/file\ with\ spaces.lisp

"hello"
admin [ /sources/gnu/clisp-clisp/build-portability2 ]$ ./clisp -norc /tmp/file\ with\ spaces.lisp

"hello"
admin [ /sources/gnu/clisp-clisp/build-portability2 ]$

[Bruno Haible]

> additional new line is printed in this case below, and I cannot know if that is a bug or not
The newline after the "hello" string does not come from your shell, it comes from clisp, function quit() in spvw.d. This is a courtesy, because most users want to see their shell prompt at the beginning of a line, and most tools that process the output of clisp expect well-terminated lines. Some implementations of 'sed', for example, have bugs when the last line of input you feed it is not terminated by a newline.

[Jean Louis]

On Sat, Nov 18, 2017 at 03:46:56PM -0000, Bruno Haible wrote:

> additional new line is printed in this case below, and I cannot know if that is a bug or not
The newline after the "hello" string does not
come from your shell, it comes from clisp,
function quit() in spvw.d. This is a courtesy,
because most users want to see their shell
prompt at the beginning of a line, and most
tools that process the output of clisp expect
well-terminated lines. Some implementations of
'sed', for example, have bugs when the last line
of input you feed it is not terminated by a
newline.

I was referring to the newline before the "hello".

What actually happened, I was testing the file
with spaces, and was in the build directory.

So I run it as:

./clisp -norc /tmp/file\ with\ spaces.lisp

and I got:

newline
newline
"hello"

while when I was running it with -q flag it was
just

newline
newline
"hello"

the ending newline I am not referencing.

However, this case is obviously repeatable only in
the build directory and not after installation of
clisp. So it is not problem.

Jean

[Jean Louis]

As I am now using new version of CLISP with the
spaces being handled, now I have problems in
runing the script that begins with:

#!/usr/bin/clisp -q -norc
(print "Hello")
(exit)

output:

GNU CLISP: invalid argument: '-q -norc'
GNU CLISP: use '-h' for help

It works from a command line.

It does not work any more from a script.

Jean

[Sam Steingold]

replace the 1st line in your script with

#!/usr/bin/clisp! -q -norc

[Jean Louis]

On Sun, Nov 19, 2017 at 05:21:20PM -0000, Sam Steingold wrote:

replace the 1st line in your script with
```

!/usr/bin/clisp! -q -norc

```

my file is named locLink

$ locLink
bash: /home/data1/protected/bin/locLink:
/usr/bin/clisp!: bad interpreter: No such file or
directory

Jean

[Sam Steingold]

you forgot to do make install

[Jean Louis]

On Sun, Nov 19, 2017 at 06:06:15PM -0000, Sam Steingold wrote:

you forgot to do make install

Of course I did that.

My clisp is installed as symlink under
/usr/bin/clisp that points to /package/prog/clisp/bin/clisp

and as such it does not work with the solution of

#!/usr/bin/clisp!

and that solution seem not reasonable to me.

--
Jean Louis

[Sam Steingold]

1. please use proper markup and the "eye" button in the upper right corner for preview.
2. the current clisp (hg tip) installs clisp! for scripting. if you do not see it, please debug why.