Thread: Recent activity and earlier message to clisp list

SourceForge Headquarters 225 Broadway Suite 1600 San Diego, CA 92101 +1 (858) 422-6466

Since there seems to be some maintenance activity, I wonder
whether there's an easy way to fix the issues reported in
https://sourceforge.net/p/clisp/mailman/message/58728542/

> whether there's an easy way to fix the issues reported in
> https://sourceforge.net/p/clisp/mailman/message/58728542/

array-dimension-limit is for general arrays.

The error message
*** - string too long: desired length 4194304 exceeds the supported maximum length
comes from this limit:

#ifdef TYPECODES
  #define stringsize_limit_1  ((uintL)(bit(intLsize-6)-1))
#else
  #define stringsize_limit_1  ((uintL)(bit(intLsize-10)-1))
#endif

The smaller of these limits is 2^22-1, that is, > 4 millions.
This is a good compromise of space (and thus speed) for functionality.
If you need larger data structures, use arrays of arrays or arrays of strings.

Bruno

Bruno Haible writes:
 > > whether there's an easy way to fix the issues reported in
 > > https://sourceforge.net/p/clisp/mailman/message/58728542/
 > 
 > array-dimension-limit is for general arrays.

I didn't realize strings were special in this sense -- 
are there other special kinds of arrays with limits smaller than
the general case?

So one possibility would be to store something other than characters
in the arrays, like char-codes of the characters.

 > The error message
 > *** - string too long: desired length 4194304 exceeds the supported maximum length
 > comes from this limit:
 > 
 > #ifdef TYPECODES
 >   #define stringsize_limit_1  ((uintL)(bit(intLsize-6)-1))
 > #else
 >   #define stringsize_limit_1  ((uintL)(bit(intLsize-10)-1))
 > #endif

So TYPECODES must be false.  
If it were true then the limit would be 16x higher.
I gather TYPECODES is controlled by some build options?  
Impnotes seems to say there are several different ways to 
get different combinations including typecodes, but I can't
tell from that which build options are available for this
platform and what other negative (or positive) consequences
would result from each.

 > The smaller of these limits is 2^22-1, that is, > 4 millions.
 > This is a good compromise of space (and thus speed) for functionality.

What's the difference between ampere and intel that leads to this
difference?
on Ampere
 cat /proc/cpuinfo
 processor       : 0
 BogoMIPS        : 50.00
 Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm lrcpc dcpop asimddp ssbs
 CPU implementer : 0x41
 CPU architecture: 8
 CPU variant     : 0x3
 CPU part        : 0xd0c
 CPU revision    : 1
and
 *features*
 (:READLINE :REGEXP :WILDCARD :SYSCALLS :I18N :LOOP :COMPILER :CLOS :MOP :CLISP :ANSI-CL :COMMON-LISP
  :LISP=CL :INTERPRETER :LOGICAL-PATHNAMES :SOCKETS :GENERIC-STREAMS :SCREEN :FFI :GETTEXT :UNICODE
  :BASE-CHAR=CHARACTER :WORD-SIZE=64 :UNIX)

 > If you need larger data structures, use arrays of arrays or arrays of strings.

not so convenient for adjustable arrays
I'm looking for alternatives to that approach.
Using a different machine seems to be another possibility.

Don Cohen wrote:
>  > array-dimension-limit is for general arrays.
> 
> I didn't realize strings were special in this sense -- 

You are aware that the string-specific limit is only minimally
smaller than the general array-dimension-limit?

> are there other special kinds of arrays with limits smaller than
> the general case?

Possibly bit-vectors; I don't remember.

> So one possibility would be to store something other than characters
> in the arrays, like char-codes of the characters.

I wouldn't recommend that. strings are optimized (vector character);
you would quite certainly lose some performance on the way.

>  > The error message
>  > *** - string too long: desired length 4194304 exceeds the supported maximum length
>  > comes from this limit:
>  > 
>  > #ifdef TYPECODES
>  >   #define stringsize_limit_1  ((uintL)(bit(intLsize-6)-1))
>  > #else
>  >   #define stringsize_limit_1  ((uintL)(bit(intLsize-10)-1))
>  > #endif
> 
> So TYPECODES must be false.  
> If it were true then the limit would be 16x higher.
> I gather TYPECODES is controlled by some build options?  
> Impnotes seems to say there are several different ways to 
> get different combinations including typecodes, but I can't
> tell from that which build options are available for this
> platform and what other negative (or positive) consequences
> would result from each.

Correct. Generally you should go with the default for the particular
platform. IIRC, there are even platforms where TYPECODES does not
work at all.

>  > The smaller of these limits is 2^22-1, that is, > 4 millions.
>  > This is a good compromise of space (and thus speed) for functionality.
> 
> What's the difference between ampere and intel that leads to this
> difference?

TYPECODES vs. not TYPECODES, I would guess?

Bruno

Bruno Haible writes:

 > You are aware that the string-specific limit is only minimally
 > smaller than the general array-dimension-limit?

I was not.
This seems problematic:
  [1]> (length (make-array (expt 2 26) ))
  Segmentation fault (core dumped)
I see now that happens also on intel:
  $ cat /proc/cpuinfo 
  processor           : 0
  vendor_id           : GenuineIntel
  cpu family          : 6
  model                 : 170
  model name            : Intel(R) Core(TM) Ultra 5 125U
  stepping              : 4
  microcode             : 0x1f

I can get up to 2^24 -1.

So what's array-total-size-limit supposed to mean?
I notice in hyperspec:
 The actual limit on the array total size imposed by the
 implementation might vary according the element type of the array; in
 this case, the value of array-total-size-limit will be the smallest
 of these possible limits.
But the current value seems to be 2^32.

 > Correct. Generally you should go with the default for the particular
 > platform. IIRC, there are even platforms where TYPECODES does not
 > work at all.
 > 
 > >  > The smaller of these limits is 2^22-1, that is, > 4 millions.
 > >  > This is a good compromise of space (and thus speed) for functionality.
 > > 
 > > What's the difference between ampere and intel that leads to this
 > > difference?

I now see that the limits look similar on the two platforms, though
I thought I used to get larger limits on intel than I see now...

Ok, this must be an older build:
  [1]> (length (make-array (1- (expt 2 24))))
  16777215
  [2]> (length (make-array (1- (expt 2 25))))
  33554431
  [3]> (length (make-array (1- (expt 2 26))))
  67108863
  [4]> (length (make-array (1- (expt 2 27))))
  Cannot map memory to address 0x334c65000 .
  [../src/spvw_mmap.d:347] errno = 12 (:ENOMEM): Cannot allocate memory.
  Trying to make room through a GC...
  *** - No more room for LISP objects
  Break 1 [5]> [6]> Bye.
  [2024-11-26 12:00:29 root@clouddb ~]
  $ cat /proc/cpuinfo 
  processor       : 0
  vendor_id       : GenuineIntel
  cpu family      : 6
  model           : 63
  model name      : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
  stepping        : 2
  microcode       : 0x49
  cpu MHz         : 2399.904
  ...
*features* on that looks the same as the one above with
  model name            : Intel(R) Core(TM) Ultra 5 125U

So now I wonder what difference between these two intel models
(or maybe the OS/libraries) causes this difference in the array
size limits and how the errors are handled.

 > TYPECODES vs. not TYPECODES, I would guess?

Is there a way to call configure to get typecodes (or at least try)?

Don Cohen wrote:
> This seems problematic:
>   [1]> (length (make-array (expt 2 26) ))
>   Segmentation fault (core dumped)

I reproduce it. With different numeric arguments, depending on the object
representation.

Bruno

Don Cohen writes:

 > So now I wonder what difference between these two intel models
 > (or maybe the OS/libraries) causes this difference in the array
 > size limits and how the errors are handled.

I now see the difference is 
CLISP 2.49.93+ (2018-02-18)
vs
CLISP 2.49+ (2010-07-17)
The newer one segfaults and limits strings to 4M.

So now the questions are
- how to figure out what memory model a given image is using
  (if that is indeed the critical difference)
- is there an easy way to build with a specific memory model ?

Don Cohen writes:
> - how to figure out what memory model a given image is using
>   (if that is indeed the critical difference)

Look at (software-type).

> - is there an easy way to build with a specific memory model ?

If you really want to go down that rabbit hole, look at the first 150 lines
of lispbibl.d.

Bruno

I'm still trying to figure out how to actually use these
"Flags that may be set through CFLAGS" - 
I'm guessing something like this
 ./configure CC='gcc -TYPECODES' build-dir 
or
 ./configure CFLAGS='-TYPECODES' build-dir
but so far I'm not guessing right.
Could I have a hint?

Bruno Haible writes:
 > Don Cohen writes:
 > > - how to figure out what memory model a given image is using
 > >   (if that is indeed the critical difference)
 > 
 > Look at (software-type).
 > 
 > > - is there an easy way to build with a specific memory model ?
 > 
 > If you really want to go down that rabbit hole, look at the first 150 lines
 > of lispbibl.d.
 > 
 > Bruno

This seems (so far) to have worked:
  ./configure CC='gcc -DTYPECODES' build-dir
then after make I get
  [1]> (software-type)
  "gcc -DTYPECODES  -g -O2 -no-integrated-cpp -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -O2 -fwrapv -fPIC -fno-strict-aliasing -DNO_ASM -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES    libgnu.a  -lreadline -lncurses /usr/local/lib/libffcall.so -Wl,-rpath -Wl,/usr/local/lib  /usr/local/lib/libsigsegv.a
  SAFETY=0 TYPECODES WIDE_HARD SPVW_BLOCKS SPVW_MIXED TRIVIALMAP_MEMORY
  libsigsegv 2.14
  libreadline 8.1
  libffcall 2.4"
  [2]> (length (make-array (* 1024 1024 5) :element-type 'character))
  5242880
  [3]> (length (make-array (* 1024 1024 9) :element-type 'character))
  9437184
  [4]> (length (make-array (* 1024 1024 17) :element-type 'character))
  17825792
  [5]> (length (make-array (* 1024 1024 33) :element-type 'character))
  34603008

Make check reports only the usual one permission denied from streams.erg

re: SAFETY=0 TYPECODES WIDE_HARD SPVW_BLOCKS SPVW_MIXED TRIVIALMAP_MEMORY
I'm a bit surprised by SAFETY=0 -- should I be worried?
The TYPECODES WIDE_HARD SPVW_BLOCKS agree with what I saw in 2.49+ (2010)
but the SPVW_MIXED TRIVIALMAP_MEMORY differs from that and is what I see
in 2.49.93+ (2018), as opposed to SPVW_PURE SINGLEMAP_MEMORY in 2.49+

I have no idea whether those choices are good, bad or indifferent.
Any advice?

Don Cohen wrote:
> This seems (so far) to have worked:
>   ./configure CC='gcc -DTYPECODES' build-dir
> then after make I get
>   [1]> (software-type)
>   "gcc -DTYPECODES  -g -O2 -no-integrated-cpp -W -Wswitch -Wcomment -Wpointer-arith -Wreturn-type -Wmissing-declarations -Wimplicit -Wno-sign-compare -Wno-format-nonliteral -Wno-shift-negative-value -O2 -fwrapv -fPIC -fno-strict-aliasing -DNO_ASM -DENABLE_UNICODE -DDYNAMIC_FFI -DDYNAMIC_MODULES    libgnu.a  -lreadline -lncurses /usr/local/lib/libffcall.so -Wl,-rpath -Wl,/usr/local/lib  /usr/local/lib/libsigsegv.a
>   SAFETY=0 TYPECODES WIDE_HARD SPVW_BLOCKS SPVW_MIXED TRIVIALMAP_MEMORY
>   libsigsegv 2.14
>   libreadline 8.1
>   libffcall 2.4"
>   [2]> (length (make-array (* 1024 1024 5) :element-type 'character))
>   5242880
>   [3]> (length (make-array (* 1024 1024 9) :element-type 'character))
>   9437184
>   [4]> (length (make-array (* 1024 1024 17) :element-type 'character))
>   17825792
>   [5]> (length (make-array (* 1024 1024 33) :element-type 'character))
>   34603008

It looks like, with this build, larger arrays are supported.

> I'm a bit surprised by SAFETY=0 -- should I be worried?

SAFETY=0 is the default setting; no need to worry.

> The TYPECODES WIDE_HARD SPVW_BLOCKS agree with what I saw in 2.49+ (2010)
> but the SPVW_MIXED TRIVIALMAP_MEMORY differs from that and is what I see
> in 2.49.93+ (2018), as opposed to SPVW_PURE SINGLEMAP_MEMORY in 2.49+
> 
> I have no idea whether those choices are good, bad or indifferent.

These are the default choices of the object representation, based on the
constraint that you want TYPECODES. The default has changed over the years,
based on practical experiences. If it works for you, all is fine.

In your build with the TYPECODES constraint, some limits may be different
(such as the max. number of conses that can be present at the same time),
and some memory/speed trade-offs may be different. But all in all, if the
only unusual requirement of yours is to have huge arrays, you can trust
this build.

Bruno

Thread: Recent activity and earlier message to clisp list

clisp-devel