clisp-devel — closed list for developers and testers

Re: DESCRIBE, impnotes, and https client

[<don...@is...>]

Bruno Haible writes:
 > We have three problems. Here's what I plan to do about them.
 > 
 > 1) The DESCRIBE function makes accesses to https://clisp.org/. This
 >    has drawbacks:
 >    - The user cannot do DESCRIBE when an internet connection is not
 >      available.

He can, but currently gets an error - perhaps that should be caught.

 >    - The documentation that the user accesses does not match the version
 >      of clisp she is using.

That could be fixed just by including the version in the url and
keeping multiple versions on the server.

 >    - https access is currently not implemented in clisp.
(discussed below)

 >    - It violates the user's privacy if other parties can infer by sniffing
 >      what documentation she is accessing.

I don't think describe should be accessing the net without 
the user explicitly asking it to, and if the user does choose to
then he can also choose whether to use http or https.

 >    Proposed solution:
 >    Let clisp use the *installed* documentation (packaged with clisp).

I don't see why it is necessary for describe to show documentation
that is not part of the run time environment.
You could supply extra functions for looking up doc and possibly
even mention them in describe output, for instance if you're 
describing an external symbol in ext or lisp packages.

 >    => No network accesses in this case.
 >    For "make check" to work before clisp is installed, add a command-line
 >    option that specifies the location of the doc. (Like option '-N'.)
 > 
 > 2) https access is currently not implemented in clisp. But the web consists
 >    of more and more https.
 >    For https, one needs SSL support with all its intricacies (TLS protocol
 >    versions, certificate checking etc.).

I've used stunnel to solve this problem for a long time.
It seems perfectly adequate and also reasonable.
I don't know whether it's supported on all clisp platforms.

There are millions of things that you MIGHT want to do that are
not built into clisp.  (Some of them are even not built into java.)
I don't see anything wrong with using some external software to do 
them from clisp.

 >    I'm no longer in favour of including something like Drakma in clisp.
 >    Instead, I believe we need to distinguish the bottom layer (SSL) and the
 >    layer sitting on top of it (https). Given SSL, https can well be done in
 >    Lisp. However, the SSL support ought to use a system-wide standard facility
 >    and the system-wide certificates (/usr/share/ca-certificates/mozilla/ on
 >    some systems) - otherwise revocations won't be handled right. This leaves us
 >    with three options:
 >      - GNUTLS,
 >      - openssl,
 >      - CL+SSL, which is based on openssl.
 > 
 >    I'll go with GNUTLS in the first place, and openssl as fallback (for
 >    portability).
 > 
 >    Since SSL cannot be applied to random streams, pipe streams, etc., but
 >    only to socket streams, the result will be a variant of SOCKET-STREAMs
 >    in clisp.
 > 
 > 3) The DESCRIBE function makes accesses to the CLHS on the web. This has 
 >    drawbacks:
 >    - The user cannot do DESCRIBE when an internet connection is not
 >      available.
 >    - It violates the user's privacy if other parties can infer by sniffing
 >      what documentation she is accessing.
 > 
 >    Proposed solution:
 >    Add a facility through which the user can download and install the
 >    complete CLHS on their system once; this requires acknowledging a license.
 >    Cf. [1]

Actually I do like that idea.
I've long kept local copies of both clhs and impnotes to view while
developing lisp code.
I've always viewed them in browsers, though, not emacs.

 >   Bruno
 > 
 > [1] https://stackoverflow.com/questions/23676164/viewing-the-common-lisp-hyperspec-offline-via-emacs

Re: DESCRIBE, impnotes, and https client

[Bruno H. <br...@cl...>]

We have three problems. Here's what I plan to do about them.

1) The DESCRIBE function makes accesses to https://clisp.org/. This
   has drawbacks:
   - The user cannot do DESCRIBE when an internet connection is not
     available.
   - The documentation that the user accesses does not match the version
     of clisp she is using.
   - https access is currently not implemented in clisp.
   - It violates the user's privacy if other parties can infer by sniffing
     what documentation she is accessing.

   Proposed solution:
   Let clisp use the *installed* documentation (packaged with clisp).
   => No network accesses in this case.
   For "make check" to work before clisp is installed, add a command-line
   option that specifies the location of the doc. (Like option '-N'.)

2) https access is currently not implemented in clisp. But the web consists
   of more and more https.
   For https, one needs SSL support with all its intricacies (TLS protocol
   versions, certificate checking etc.).

   I'm no longer in favour of including something like Drakma in clisp.
   Instead, I believe we need to distinguish the bottom layer (SSL) and the
   layer sitting on top of it (https). Given SSL, https can well be done in
   Lisp. However, the SSL support ought to use a system-wide standard facility
   and the system-wide certificates (/usr/share/ca-certificates/mozilla/ on
   some systems) - otherwise revocations won't be handled right. This leaves us
   with three options:
     - GNUTLS,
     - openssl,
     - CL+SSL, which is based on openssl.

   I'll go with GNUTLS in the first place, and openssl as fallback (for
   portability).

   Since SSL cannot be applied to random streams, pipe streams, etc., but
   only to socket streams, the result will be a variant of SOCKET-STREAMs
   in clisp.

3) The DESCRIBE function makes accesses to the CLHS on the web. This has 
   drawbacks:
   - The user cannot do DESCRIBE when an internet connection is not
     available.
   - It violates the user's privacy if other parties can infer by sniffing
     what documentation she is accessing.

   Proposed solution:
   Add a facility through which the user can download and install the
   complete CLHS on their system once; this requires acknowledging a license.
   Cf. [1]

Comments?

  Bruno

[1] https://stackoverflow.com/questions/23676164/viewing-the-common-lisp-hyperspec-offline-via-emacs

Re: Use (f)puts instead of (f)printf when possible.

[Sam S. <sd...@gn...>]

Hi Bruno,

> * Bruno Haible <oe...@py...t> [2017-09-10 16:20:26 +0200]:
>
>> > #define fprint(fp,string) fputs(string,fp)
>> > #define print(string) fputs(string,stdout)
>> 
>> I don't like this for 2 reasons:
>> 
>> 2. I prefer names that are not substrings of other names for the sake of
>> grepping convenience. E.g., I would use `cprint` instead of `fprint' and
>> `cputs` instead of `print` ("c" stands for CLISP &c)
>
> You can use the grep option '-w' to search for 'print' only:
>   $ grep -w print *.d
>   $ grep -w 'print *(' *.d

Yes, I can also use regexps and tag-search in emacs.
This is not the point.
The point is not "impossible".
The point is "I need to make an extra effort".

"Write" (code) is done once.
"Read" is done many times.

Replacing "print" with, e.g., "cprint" is done _once_ by _one_ person.
The benefits are reaped _every_ time _anyone_ is reading the code.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://camera.org
http://americancensorship.org http://www.memritv.org http://www.dhimmitude.org
Nostalgia isn't what it used to be.

Re: Use (f)puts instead of (f)printf when possible.

[Bruno H. <br...@cl...>]

Hi Sam,

> > #define fprint(fp,string) fputs(string,fp)
> > #define print(string) fputs(string,stdout)
> 
> I don't like this for 2 reasons:
> 
> 1. I am opposed to gratuitous deviations from the standard coding
> practices because they raise the barrier to entry for new people.

I understand the "barrier to entry for new people" argument. But in this
case I found it worth it.

> 2. I prefer names that are not substrings of other names for the sake of
> grepping convenience. E.g., I would use `cprint` instead of `fprint' and
> `cputs` instead of `print` ("c" stands for CLISP &c)

You can use the grep option '-w' to search for 'print' only:
  $ grep -w print *.d
  $ grep -w 'print *(' *.d

Bruno

Re: Use (f)puts instead of (f)printf when possible.

[Bruno H. <br...@cl...>]

Jörg wrote on 2017-08-25:
> 1) There is an (undocumented) way to avoid this security warning.
> Test case:
> ===============================================================================
> #include <stdio.h>
> extern const char * transform1 (const char * s);
> extern const char * transform2 (const char * s) __attribute__ ((__format_arg__ (1)));
> void foo1 () { fprintf(stderr, transform1("Hello")); }
> void foo2 () { fprintf(stderr, transform2("Hello")); }
> ===============================================================================
> $ gcc -S -Wall -Wformat-security foo.c
> foo.c: In function 'foo1':
> foo.c:4:1: warning: format not a string literal and no format arguments [-Wformat-security]
>  void foo1 () { fprintf(stderr, transform1("Hello")); }
>  ^~~~
> As you can see, this __attribute__ ((__format_arg__ (1))) has the effect of
> avoiding the warning.
> -----
> 
> What is undocumented?
> https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html
> https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Function-Attributes.html
> ... mentions __attribute__ ((format_arg...)).

It is not documented that this attribute will silence a -Wformat-security
warning.

> here I find that declaration completely unsafe, because it depends on foreign PO/MO files' contents!!

GCC itself declares the gettext function like this.
builtins.def [1] contains this:

  DEF_EXT_LIB_BUILTIN    (BUILT_IN_GETTEXT, "gettext", BT_FN_STRING_CONST_STRING, ATTR_FORMAT_ARG_1)

The ATTR_FORMAT_ARG_1 is __attribute__ ((__format_arg__ (1))).

clisp does the same with gettext, clgettext, clgettext1 (which are merely
variants of gettext).

So, apparently the GCC developers think it's not worth for GCC to complain
about every use of a translated format string. The reason is that the
gettext tools ('msgfmt' in particular) contain the appropriate checking.

Bruno

[1] https://gcc.gnu.org/viewcvs/gcc/trunk/gcc/builtins.def?revision=249685&view=co

Re: Use (f)puts instead of (f)printf when possible.

[Bruno H. <br...@cl...>]

Pascal Bourguignon wrote on 2017-08-25:
> >      fprint(stderr,GETTEXTL(“Warning: “));
> 
> The point is that this is some external data, obtained at run-time from files that are not necessarily under the control of the program (eg. some distributions package the localisable file separately).

'fprint' does not do format string processing (unlike fprintf).
Therefore if some malicious translation is present in a .mo file, the
program will output it, but it will not crash from it. (Although if
it contains ANSI control codes, it may make your terminal go into
strange states or crash. But that's a bug in the terminal emulator.)

The security risks for setuid programs are handled through glibc/intl/
or libintl (on non-glibc systems).

Bruno

Re: Use (f)puts instead of (f)printf when possible.

[Sam S. <sd...@gn...>]

> * Pascal Bourguignon <cwo@vasbezngvzntb.pbz> [2017-08-25 20:07:30 +0200]:
>
>> On 25 Aug 2017, at 08:50, Bruno Haible <br...@cl...> wrote:
>> 
>> Pascal Bourguignon wrote:
>>> The standard way to avoid it is to use a literal format string!
>>> 
>>>    fprintf(stderr,”%s”,GETTEXTL(“Warning: “));
>> 
>> Indeed gcc recognizes "%s" (and "%c") as special cases of format strings
>> and optimizes these cases. But I still find such code harder to read than
>> 
>>      fprint(stderr,GETTEXTL(“Warning: “));
>
>
> The point is that this is some external data, obtained at run-time from
> files that are not necessarily under the control of the program
> (eg. some distributions package the localisable file separately).
>
> Therefore if you insist on avoiding “%s” you should, to keep the program
> correct and secure, do the following:
>
>      fprint(stderr,escape_print_directives(GETTEXTL(“Warning: “)));
>
> and suddenly, using “%s” looks way simpler.

This only avoids a minor problem.
What would you do with

    fprintf(stderr,GETTEXTL("OS Error: %s %d"),a,b)

In fact there is no real problem here: GETTEXTL _is_ under our control,
it takes the data from our translation files.

Thanks

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://iris.org.il
http://think-israel.org http://islamexposedonline.com http://jij.org
Democracy vs Despotism = Theft vs Robbery

Re: Use (f)puts instead of (f)printf when possible.

[Pascal B. <pj...@in...>]

> On 25 Aug 2017, at 08:50, Bruno Haible <br...@cl...> wrote:
> 
> Pascal Bourguignon wrote:
>> The standard way to avoid it is to use a literal format string!
>> 
>>    fprintf(stderr,”%s”,GETTEXTL(“Warning: “));
> 
> Indeed gcc recognizes "%s" (and "%c") as special cases of format strings
> and optimizes these cases. But I still find such code harder to read than
> 
>      fprint(stderr,GETTEXTL(“Warning: “));


The point is that this is some external data, obtained at run-time from files that are not necessarily under the control of the program (eg. some distributions package the localisable file separately).

Therefore if you insist on avoiding “%s” you should, to keep the program correct and secure, do the following:

     fprint(stderr,escape_print_directives(GETTEXTL(“Warning: “)));

and suddenly, using “%s” looks way simpler.


-- 
__Pascal J. Bourguignon__

Re: Use (f)puts instead of (f)printf when possible.

[Sam S. <sd...@gn...>]

> * Bruno Haible <oe...@py...t> [2017-08-24 23:11:32 +0200]:
>
> 2) I dislike APIs that are hard to remember.

No one likes them.
However, as long as they are commonly known and accepted,
deviation from them is counterproductive.

There are many such idiosyncrasies.
E.g., omitting multiplication signs and parentheses in math.
The optimal solution is a complete switch from infix to prefix notation.
Do you think this will ever happen?
Do you think using the infix notation in CLISP C files and adding a
pre-processor converting the prefix notation to infix is a good idea?

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net https://jihadwatch.org
http://mideasttruth.com http://www.memritv.org http://camera.org
Sniper's job is to reduce his target audience.

Re: Use (f)puts instead of (f)printf when possible.

[Sam S. <sd...@gn...>]

Hi Bruno,

> * Bruno Haible <oe...@py...t> [2017-08-24 23:11:32 +0200]:
>
> #define fprint(fp,string) fputs(string,fp)
> #define print(string) fputs(string,stdout)

I don't like this for 2 reasons:

1. I am opposed to gratuitous deviations from the standard coding
practices because they raise the barrier to entry for new people.

2. I prefer names that are not substrings of other names for the sake of
grepping convenience. E.g., I would use `cprint` instead of `fprint' and
`cputs` instead of `print` ("c" stands for CLISP &c)

Thus I am with Pascal and Joerg on this.

However, I don't think this is important enough to warrant further
discussion.

Thanks.

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net http://honestreporting.com
https://jihadwatch.org http://think-israel.org http://americancensorship.org
If You Want Breakfast In Bed, Sleep In the Kitchen.

Re: Use (f)puts instead of (f)printf when possible.

[<Joe...@t-...>]

Hi,

Bruno wrote:
-----
! fprintf(stderr, GETTEXTL("Warning: "));
! fputs(GETTEXTL("Warning: "),stderr);
1) There is an (undocumented) way to avoid this security warning.
Test case:
===============================================================================
#include <stdio.h>
extern const char * transform1 (const char * s);
extern const char * transform2 (const char * s) __attribute__ ((__format_arg__ (1)));
void foo1 () { fprintf(stderr, transform1("Hello")); }
void foo2 () { fprintf(stderr, transform2("Hello")); }
===============================================================================
$ gcc -S -Wall -Wformat-security foo.c
foo.c: In function 'foo1':
foo.c:4:1: warning: format not a string literal and no format arguments [-Wformat-security]
 void foo1 () { fprintf(stderr, transform1("Hello")); }
 ^~~~
As you can see, this __attribute__ ((__format_arg__ (1))) has the effect of
avoiding the warning.
-----

What is undocumented?
https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html
https://gcc.gnu.org/onlinedocs/gcc-3.1/gcc/Function-Attributes.html
... mentions __attribute__ ((format_arg...)).

Is the side-effect undocumented? But that's specifically the intention of using such an attribute:
Have the compiler know & check where format strings are, or let the programmer declaim that the function
is safe -- here I find that declaration completely unsafe, because it depends on foreign PO/MO files' contents!!

Funny, I remember attribute((format)), but format_arg was new to me today.

Summary: either
- use %s with printf, or
- avoid %s and printf altogether and instead use fputs.

Regards,
	Jörg

Re: Trouble building 2.49.60 beta on Fedora

[Bruno H. <br...@cl...>]

Hi Jerry,

> Fedora builds with -Werror=format-security, so I had to change
> one line in the aarch64 patch.  That patch modifies spvw_allocate.d
> and in one place issues a warning.  I had to change this line:
> 
> fprintf(stderr, GETTEXTL("Warning: "));
> 
> to:
> 
> fputs(GETTEXTL("Warning: "),stderr);
> 
> since the string is non-constant, and therefore triggers the format
> security error.

Thanks for the heads-up. We have now added two workarounds; each of them
would be sufficient to avoid this warning.

Bruno

Re: Use (f)puts instead of (f)printf when possible.

[Bruno H. <br...@cl...>]

Pascal Bourguignon wrote:
> The standard way to avoid it is to use a literal format string!
> 
>     fprintf(stderr,”%s”,GETTEXTL(“Warning: “));

Indeed gcc recognizes "%s" (and "%c") as special cases of format strings
and optimizes these cases. But I still find such code harder to read than

      fprint(stderr,GETTEXTL(“Warning: “));

Bruno

Re: Use (f)puts instead of (f)printf when possible.

[Pascal B. <pj...@in...>]

> On 24 Aug 2017, at 23:11, Bruno Haible <br...@cl...> wrote:
> 
> Hi Sam,
> 
>> Use (f)puts instead of (f)printf when possible.
> 
> I guess this was motivated by Jerry's remark:
> 
> ! Fedora builds with -Werror=format-security, so I had to change
> ! one line in the aarch64 patch.  That patch modifies spvw_allocate.d
> ! and in one place issues a warning.  I had to change this line:
> ! 
> ! fprintf(stderr, GETTEXTL("Warning: “));
> ! 
> ! to:
> ! 
> ! fputs(GETTEXTL("Warning: "),stderr);
> ! 
> ! since the string is non-constant, and therefore triggers the format
> ! security error.
> 

The standard way to avoid it is to use a literal format string!

    fprintf(stderr,”%s”,GETTEXTL(“Warning: “));


> Therefore I've introduced two macros
> 
> /* Use fprintf and printf only for format strings that take at least 1 argument.
>   For literal strings, use print and fprint.
>   Avoid using fputs, puts, fputc, putc, putchar directly, because these APIs
>   are hard to memorize (fputs, fputc, putc don't take the stream first; puts
>   outputs an extra newline) or redundant (fputc, putc, putchar are special
>   cases of fputs that GCC recognizes anyway). */
> #define fprint(fp,string) fputs(string,fp)
> #define print(string) fputs(string,stdout)

At first, I thought that fprint would take a format string (why fprint(fp,“3 %%”); prints two percent characters?).

At second, it feels like a typo!

I would prefer fputs...

-- 
__Pascal J. Bourguignon__

Re: Use (f)puts instead of (f)printf when possible.

[Bruno H. <br...@cl...>]

Hi Sam,

> Use (f)puts instead of (f)printf when possible.

I guess this was motivated by Jerry's remark:

! Fedora builds with -Werror=format-security, so I had to change
! one line in the aarch64 patch.  That patch modifies spvw_allocate.d
! and in one place issues a warning.  I had to change this line:
! 
! fprintf(stderr, GETTEXTL("Warning: "));
! 
! to:
! 
! fputs(GETTEXTL("Warning: "),stderr);
! 
! since the string is non-constant, and therefore triggers the format
! security error.

Two things about this:

1) There is an (undocumented) way to avoid this security warning.

Test case:
===============================================================================
#include <stdio.h>
extern const char * transform1 (const char * s);
extern const char * transform2 (const char * s) __attribute__ ((__format_arg__ (1)));
void foo1 () { fprintf(stderr, transform1("Hello")); }
void foo2 () { fprintf(stderr, transform2("Hello")); }
===============================================================================
$ gcc -S -Wall -Wformat-security foo.c
foo.c: In function 'foo1':
foo.c:4:1: warning: format not a string literal and no format arguments [-Wformat-security]
 void foo1 () { fprintf(stderr, transform1("Hello")); }
 ^~~~

As you can see, this __attribute__ ((__format_arg__ (1))) has the effect of
avoiding the warning.

But since it is undocumented, I agree to avoid fprintf with format-strings
that take 0 arguments, and use literal strings instead. Even though it will
likely cause work for the translators.

2) I dislike APIs that are hard to remember.
   * fputs, fputc, putc have the problem that it requires you do put the stream
     after the arguments, which is unlike fprintf, object_out, etc.
   * puts additionally emits an extra newline.
   Really, when writing a sequence of outputs to the same stream, these
   inconsistencies lead to mistakes: extra or missing newlines at the end of
   an output.
   If we were using <stdio.h> only in 1 or 2 places, this would be acceptable.
   But we're using it all over the place.

   Additionally, there is no need to optimize
     fputs("x",stdout) to fputc('x',stdout)
   because the compiler (gcc) does this optimization already for years.

Therefore I've introduced two macros

/* Use fprintf and printf only for format strings that take at least 1 argument.
   For literal strings, use print and fprint.
   Avoid using fputs, puts, fputc, putc, putchar directly, because these APIs
   are hard to memorize (fputs, fputc, putc don't take the stream first; puts
   outputs an extra newline) or redundant (fputc, putc, putchar are special
   cases of fputs that GCC recognizes anyway). */
#define fprint(fp,string) fputs(string,fp)
#define print(string) fputs(string,stdout)

The code looks now more consistent, since there are now exactly 4 primitives:
  fprintf
  printf
  fprint
  print
and they have the signature that you expect.

Bruno

Re: (format ... "~A" (apply #'format nil errorstring arguments))

[Sam S. <sd...@gn...>]

> * Bruno Haible <oe...@py...t> [2017-08-23 22:09:17 +0200]:
>
> Sam wrote on 2017-08-02:
>> In clos-method2.lisp and many other places you have this pattern:
>> 
>> --8<---------------cut here---------------start------------->8---
>> (defmacro program-error-reporter (caller)
>>   `#'(lambda (form detail errorstring &rest arguments)
>>        (apply #'sys::lambda-list-error form detail
>>               "~S: ~A" ,caller (apply #'format nil errorstring arguments))))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> Why not
>> 
>> --8<---------------cut here---------------start------------->8---
>> (defmacro program-error-reporter (caller)
>>   `#'(lambda (form detail errorstring &rest arguments)
>>        (apply #'sys::lambda-list-error form detail
>>               "~S: ~?" ,caller errorstring arguments)))
>> --8<---------------cut here---------------end--------------->8---
>> 
>> The only place where the second pattern is used is invalid-method-error
>> and method-combination-error in close-methcomb2.
>
> I replied:
>> Both are equivalent.
>
> Not always. When the format string contains directives that are
> sensitive on the current column position (tabulate, justification, all
> pretty-printing directives) or newline state (fresh-line,
> elastic-newline), the two idioms may produce different results.

Valid point, and, I think, an excellent argument in favor of the second
version.

Thanks

-- 
Sam Steingold (http://sds.podval.org/) on darwin Ns 10.3.1504
http://steingoldpsychology.com http://www.childpsy.net
http://honestreporting.com http://mideasttruth.com
MS DOS: Keyboard not found. Press F1 to continue.

Re: (format ... "~A" (apply #'format nil errorstring arguments))

[Bruno H. <br...@cl...>]

Sam wrote on 2017-08-02:
> In clos-method2.lisp and many other places you have this pattern:
> 
> --8<---------------cut here---------------start------------->8---
> (defmacro program-error-reporter (caller)
>   `#'(lambda (form detail errorstring &rest arguments)
>        (apply #'sys::lambda-list-error form detail
>               "~S: ~A" ,caller (apply #'format nil errorstring arguments))))
> --8<---------------cut here---------------end--------------->8---
> 
> Why not
> 
> --8<---------------cut here---------------start------------->8---
> (defmacro program-error-reporter (caller)
>   `#'(lambda (form detail errorstring &rest arguments)
>        (apply #'sys::lambda-list-error form detail
>               "~S: ~?" ,caller errorstring arguments)))
> --8<---------------cut here---------------end--------------->8---
> 
> The only place where the second pattern is used is invalid-method-error
> and method-combination-error in close-methcomb2.

I replied:
> Both are equivalent.

Not always. When the format string contains directives that are sensitive
on the current column position (tabulate, justification, all pretty-printing
directives) or newline state (fresh-line, elastic-newline), the two idioms
may produce different results.

I don't know whether this has an impact on your change from 2017-08-02.

Bruno

Re: Trouble building 2.49.60 beta on Fedora

[Bruno H. <br...@cl...>]

Hi Jerry,

> Thanks again, Bruno, for the patches.  They don't apply cleanly to
> either the 2.49.60 beta release or to current mercurial head, though.
> What version of the code are those patches against?

The patches were meant to apply to the 2.49.60 beta release. They do
apply without rejects, if you apply them in the order I sent them:

$ cd clisp-2.49.60/
$ patch -p1 < ../gdbm.diff
patching file modules/gdbm/gdbm.c
$ patch -p1 < ../arm64.diff 
patching file src/lispbibl.d
Hunk #1 succeeded at 390 (offset -4 lines).
Hunk #2 succeeded at 2898 (offset -4 lines).
Hunk #3 succeeded at 3029 (offset -4 lines).
Hunk #4 succeeded at 3048 (offset -4 lines).
Hunk #5 succeeded at 3068 (offset -4 lines).
Hunk #6 succeeded at 3966 (offset -4 lines).
patching file src/makemake.in
patching file src/spvw.d
patching file src/spvw_allocate.d
$ patch -p1 < ../powerpc64.diff 
patching file src/lispbibl.d
Hunk #1 succeeded at 2917 (offset -4 lines).
Hunk #2 succeeded at 3406 (offset -4 lines).
patching file src/spvw.d
$ patch -p1 < ../s390x.diff 
patching file src/lispbibl.d
Hunk #1 succeeded at 2378 (offset -4 lines).
Hunk #2 succeeded at 2388 (offset -4 lines).
Hunk #3 succeeded at 2420 (offset -4 lines).
Hunk #4 succeeded at 2621 (offset -4 lines).
Hunk #5 succeeded at 2942 (offset -4 lines).
patching file src/spvw.d

> Sadly, the patches have not improved the situation.  The 32- and
> 64-bit x86 and 32-bit ARM builds still succeed, and all of the others
> segfault now.
> That build was against mercurial revision 820ad4b724cb, with the
> following patches: ...

Please use the 2.49.60 beta release as a starting point. If you use
the mercurial HEAD, you live more dangerously. There are some kinds of
bugs that we detect and fix using special testing modes during beta testing,
and this has not yet happened for the changes between 2.49.60 and HEAD.

> - A patch to fix the ARM build:
> http://sourceforge.net/tracker/?func=detail&aid=3529615&group_id=1355&atid=301355
> - A patch to fix configure detection of Berkeley DB:
> http://sourceforge.net/tracker/?func=detail&aid=3572516&group_id=1355&atid=301355
> - A patch to add definitions for ERFKILL and EHWPOISON, and to remove
> the definition of cfree, which has been dropped from recent versions
> of glibc (attached).
> - The gdbm patch I sent with the initial email

These patches look all reasonable. They are surely not the causes of crashes
on Linux/x86_64.

If you still get segfaults on all other platforms, it may be due to
  * optimization bugs in the GCC you are using: please patch makemake.in
    to use only -O instead of -O2 and try that,
  * different behaviour of the Linux kernel. There is a crude workaround,
    which is to configure with --enable-portability, but this disables
    a lot of optimization features (including generational GC), therefore
    this is not the first thing I'd recommend to you. The other approach
    would be for me to debug it in a VM for x86_64 or i386. Which Fedora
    installer image is it that I can download and try in a VM?

> I just noticed that we add these flags when building for ppc64/ppc64le:
> 
> -DNO_GENERATIONAL_GC -DNO_MULTIMAP_FILE -DNO_SINGLEMAP
> 
> That part of the spec file predates my involvement, so I don't know
> why they are in there.  Should we be using those flags?

I tested the powerpc64.diff without these options, therefore I'm pretty
sure you can remove them.

Bruno

Re: Trouble building 2.49.60 beta on Fedora

[Bruno H. <br...@cl...>]

Jerry James wrote:
> the s390x, ppc64 and ppc64le builds all fail with this message:
> 
> ../src/lispbibl.d:2987:2: error: #error CLISP has not been ported to
> this platform - oint_type_len undefined
>  #error CLISP has not been ported to this platform - oint_type_len undefined

For s390x support, please try this patch.

Bruno

Re: Trouble building 2.49.60 beta on Fedora

[Jerry J. <log...@gm...>]

Thanks again, Bruno, for the patches.  They don't apply cleanly to
either the 2.49.60 beta release or to current mercurial head, though.
What version of the code are those patches against?

Sadly, the patches have not improved the situation.  The 32- and
64-bit x86 and 32-bit ARM builds still succeed, and all of the others
segfault now.  I'm not sure if you can follow this link or not, but if
you can, build logs will be here for a few days:

https://koji.fedoraproject.org/koji/taskinfo?taskID=21412890

That build was against mercurial revision 820ad4b724cb, with the
following patches:

- A patch to fix the ARM build:
http://sourceforge.net/tracker/?func=detail&aid=3529615&group_id=1355&atid=301355
- A patch to fix configure detection of Berkeley DB:
http://sourceforge.net/tracker/?func=detail&aid=3572516&group_id=1355&atid=301355
- A patch to add definitions for ERFKILL and EHWPOISON, and to remove
the definition of cfree, which has been dropped from recent versions
of glibc (attached).
- The gdbm patch I sent with the initial email
- The ppc64 and aarch64 patches you have sent in this thread, adapted
to apply to the mercurial revision noted above.

Also, Fedora builds with -Werror=format-security, so I had to change
one line in the aarch64 patch.  That patch modifies spvw_allocate.d
and in one place issues a warning.  I had to change this line:

fprintf(stderr, GETTEXTL("Warning: "));

to:

fputs(GETTEXTL("Warning: "),stderr);

since the string is non-constant, and therefore triggers the format
security error.

I just noticed that we add these flags when building for ppc64/ppc64le:

-DNO_GENERATIONAL_GC -DNO_MULTIMAP_FILE -DNO_SINGLEMAP

That part of the spec file predates my involvement, so I don't know
why they are in there.  Should we be using those flags?

Please let me know if there is anything I can do to help debug these
segfaults further.  Regards,
-- 
Jerry James
http://www.jamezone.org/

Re: https client

[Bruno H. <br...@cl...>]

Lucas Buchala wrote:
> It seems to expect an URL with prefix "http://" but a secure URL with
> prefix "https://" is found in the "Location:" header instead, so it
> ends up concatenating the strings:

And clisp does not yet have a https client.

More and more of the web switches over from http to https. It was easy to
write an http client in a couple of lines of code (like in clhs.lisp).
But for https, one needs SSL support with all its intricacies (TLS protocol
versions, certificate checking etc.).

Anyone wants to contribute a clisp module that implements a http/https client?
It's not needed to write the code from scratch; such code already exists
(under usable licenses).

According to [1], the most compelling choices would be:
  * drakma
  * drakma-async, based on cl-async, itself based on libuv.

Any takers?

Bruno

[1] http://www.cliki.net/HTTP%20client

Re: DESCRIBE broken after website move

[Pascal B. <pj...@in...>]

> On 22 Aug 2017, at 16:13, Lucas Buchala <luc...@gm...> wrote:
> 
> On Mon, Aug 21, 2017 at 9:47 PM, Bruno Haible <br...@cl...> wrote:
>> As a consequence of the move from http://clisp.sourceforge.net/
>> to https://clisp.sourceforge.io/ the function DESCRIBE is now broken:
> 
> Hello. I'm newbie in Lisp, but this error looked interesting, so I
> went to take a look at src/clhs.lisp.
> 
> I wonder if there's some problem in the function open-http, around line 198:
> https://sourceforge.net/p/clisp/clisp/ci/default/tree/src/clhs.lisp#l198
> 
> It seems to expect an URL with prefix "http://" but a secure URL with
> prefix "https://" is found in the "Location:" header instead, so it
> ends up concatenating the strings:
> 
>  (unless (string-equal #1# new-url
>                        :end2 (min (length new-url) #2#))
>    (setq new-url (string-concat #1# host new-url)))


Indeed, instead it should test for urls with a scheme prefix! (Any scheme!)

Schemes are [A-Za-z][-+.A-Za-z0-9]*
followed by a colon.

So if the string matches ^[A-Za-z][-+.A-Za-z0-9]*:
then it contains a scheme and it should not be considered a path, but a whole URI.

https://en.wikipedia.org/wiki/Uniform_Resource_Identifier
-- 
__Pascal J. Bourguignon__

Re: DESCRIBE broken after website move

[Lucas B. <luc...@gm...>]

On Mon, Aug 21, 2017 at 9:47 PM, Bruno Haible <br...@cl...> wrote:
> As a consequence of the move from http://clisp.sourceforge.net/
> to https://clisp.sourceforge.io/ the function DESCRIBE is now broken:

Hello. I'm newbie in Lisp, but this error looked interesting, so I
went to take a look at src/clhs.lisp.

I wonder if there's some problem in the function open-http, around line 198:
https://sourceforge.net/p/clisp/clisp/ci/default/tree/src/clhs.lisp#l198

It seems to expect an URL with prefix "http://" but a secure URL with
prefix "https://" is found in the "Location:" header instead, so it
ends up concatenating the strings:

  (unless (string-equal #1# new-url
                        :end2 (min (length new-url) #2#))
    (setq new-url (string-concat #1# host new-url)))

Re: Trouble building 2.49.60 beta on Fedora - s390x

[Jerry J. <log...@gm...>]

On Mon, Aug 21, 2017 at 12:24 AM, Bruno Haible <br...@cl...> wrote:
> I am testing a patch for s390x support. But my build machine is quite slow [1],
> therefore this testing takes 24 hours.

Yes, QEMU emulation is quite slow.  I did not mean to sound impatient.
Anyway, I have encountered some technical difficulties tonight,
unrelated to clisp, which have prevented me from making progress.  I
will try again tomorrow.

Regards,
-- 
Jerry James
http://www.jamezone.org/

DESCRIBE broken after website move

[Bruno H. <br...@cl...>]

As a consequence of the move from http://clisp.sourceforge.net/
to https://clisp.sourceforge.io/ the function DESCRIBE is now broken:

[1]> (describe nil)

NIL is the empty list, the symbol NIL, lies in #<PACKAGE COMMON-LISP>, is accessible in 11 packages CLOS, COMMON-LISP, COMMON-LISP-USER, EXPORTING, EXT,
FFI, POSIX, READLINE, REGEXP, SCREEN, SYSTEM, a constant, value: NIL, names a type, names a built-in foreign type, has 2 properties SYSTEM::INSTRUCTION,
SYSTEM::TYPE-SYMBOL
;; connecting to "http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Data/Map_Sym.txt"...connected...HTTP/1.1 404 Not Found
;; "Date: Tue, 22 Aug 2017 00:39:06 GMT"
;; "Server: Apache/2.4.7 (Ubuntu)"
;; "Content-Length: 253"
;; "Connection: close"
;; "Content-Type: text/html; charset=iso-8859-1"
;; ""
;; "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">"
;; "<html><head>"
;; "<title>404 Not Found</title>"
;; "</head><body>"
;; "<h1>Not Found</h1>"
;; "<p>The requested URL /projects/iiip/doc/CommonLISP/HyperSpec/Data/Map_Sym.txt was not found on this server.</p>"
;; "</body></html>"
;; connecting to "http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Data/Symbol-Table.text"...connected...HTTP/1.1 200 OK...45,322 bytes
;; SYSTEM::GET-CLHS-MAP(#<IO INPUT-BUFFERED SOCKET-STREAM CHARACTER www.ai.mit.edu:80>)...978/978 symbols
.
ANSI-CL Documentation is at
"http://www.ai.mit.edu/projects/iiip/doc/CommonLISP/HyperSpec/Body/any_nil.html"
;; connecting to "http://clisp.org/impnotes/id-href.map"...connected...HTTP/1.1 301 Moved Permanently --> "https://clisp.sourceforge.io/impnotes/id-href.map"
;; connecting to "http://clisp.orghttps://clisp.sourceforge.io/impnotes/id-href.map"...
*** - PARSE-INTEGER: substring "" does not have integer syntax at position 0
The following restarts are available:
ABORT          :R1      Abort main loop

It looks like an invalid URL
"http://clisp.orghttps://clisp.sourceforge.io/impnotes/id-href.map" was constructed
(maybe in clhs.lisp?).

Bruno