From: John K. <jk2...@ya...> - 2004-04-24 03:20:12
|
Apologies if this is a thick question, but what is the maximum memory usage allowed? On OS X 10.3.3 in SBCL 0.8.8 and 0.8.9 I am trying: (let ((f (make-array '(4096 4096) :element-type 'double-float))) (= (aref f 10 11) 99d0)) ;; this line is to access the array & return bool This should allocate 2^12*2^12*2^3=2^27 bytes, which is about 135 MB, which I thought would be within the addressable space. The machine has 2 GB of RAM, and system limits are given at bottom. But when I do this on 0.8.8 I get the error set_auto_gc_trigger: tried to set gc trigger too high! (0x853bc5c) fatal error encountered in SBCL pid 2373: lost There's no LDB in this build; exiting. and in 0.8.9 I find that the next (gc :full t) fails with fatal error encountered in SBCL pid 2399: GC invariant lost, file "gc-common.c", line 612 The system is too badly corrupted or confused to continue at the Lisp Many thanks in advance. JK ================================== usage limits as given in tcsh sbcl-0.8.9-ppc-darwin> limit cputime unlimited filesize unlimited datasize 750000 kbytes stacksize 40960 kbytes coredumpsize 20000000 kbytes memoryuse unlimited descriptors 256 memorylocked unlimited maxproc 532 __________________________________ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash |
From: David S. <da...@da...> - 2004-04-24 07:17:21
|
John Klein <jk2...@ya...> writes: > Apologies if this is a thick question, but what is the maximum memory > usage allowed? > > On OS X 10.3.3 in SBCL 0.8.8 and 0.8.9 I am trying: I think the VM gives you ~2GB. You'll have to check that. > (let ((f (make-array '(4096 4096) :element-type 'double-float))) > (= (aref f 10 11) 99d0)) ;; this line is to access the array & return bool I can reproduce your bug. (Is that really an SBCL bug? OpenMCL also blows up if you try the above.) OpenMCL doesn't give you as big an array size limit as SBCL. Here is a work around for you: $ sbcl This is SBCL 0.8.9.35, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * (describe 'ARRAY-TOTAL-SIZE-LIMIT) ARRAY-TOTAL-SIZE-LIMIT is an external symbol in #<PACKAGE "COMMON-LISP">. It is a constant; its value is 536870911. Constant documentation: the exclusive upper bound on the total number of elements in an array * (let ((f (make-array '(4096 4096) :initial-element 1d0))) (= (aref f 10 11) 99d0)) NIL This works in just a couple seconds on my machine. Tell you what. I'll do an exact time: * (time (let ((f (make-array '(4096 4096) :initial-element 1d0))) (= (aref f 10 11) 99d0))) Illegal instruction Heh. Oops! Let's try again. Restart sbcl... * (time (let ((f (make-array '(4096 4096) :initial-element 1d0))) (= (aref f 10 11) 99d0))) Evaluation took: 5.929 seconds of real time 4.62 seconds of user run time 0.55 seconds of system run time 0 page faults and 67109088 bytes consed. NIL If I specify :element-type 'double-float, I find myself spending all my CPU time in system and I get too impatient to let it finish, leading me to force quit sbcl. BTW, I tried this in OpenMCL and this is what happened: $ openmcl Welcome to OpenMCL Version (Beta: Darwin) 0.14.1-p1! ? (time (let ((f (make-array '(4096 4096) :initial-element 1d0))) (= (aref f 10 11) 99d0))) > Error in process listener(1): value 16777216 is not of the expected type (UNSIGNED-BYTE 24). > While executing: CCL::MAKE-UARRAY-1 > Type :POP to abort. Type :? for other options. 1 > :pop ? (describe 'ARRAY-TOTAL-SIZE-LIMIT) Symbol: ARRAY-TOTAL-SIZE-LIMIT Constant EXTERNAL in package: #<Package "COMMON-LISP"> Print name: "ARRAY-TOTAL-SIZE-LIMIT" Value: 16777216 Function: #<Unbound> Plist: NIL HTH -- I wouldn't mind the rat race so much if it wasn't for all the damn cats. |
From: John K. <jk2...@ya...> - 2004-04-24 08:21:11
|
Just to see if I understand where things stand... --- David Steuber <da...@da...> wrote: > > I can reproduce your bug. (Is that really an SBCL bug? OpenMCL also > blows up if you try the above.) OpenMCL doesn't give you as big an > array size limit as SBCL. Here is a work around for you: > > ARRAY-TOTAL-SIZE-LIMIT is an external symbol in #<PACKAGE "COMMON-LISP">. > It is a constant; its value is 536870911. so ARRAY-TOTAL-SIZE-LIMIT is much smaller than the 10^24 elements in my array. And SBCL shouldn't die when allocating an excessively big array, I think. > > * (let ((f (make-array '(4096 4096) :initial-element 1d0))) > (= (aref f 10 11) 99d0)) > > This works in just a couple seconds on my machine. Tell you what. > I'll do an exact time: OK, but I _think_ this makes an array of 32 bit pointers pointing to the same boxed double float. So I think it is half as big as the 'double-float array. > BTW, I tried this in OpenMCL and this is what happened: > > $ openmcl > Welcome to OpenMCL Version (Beta: Darwin) 0.14.1-p1! ..... > ? (describe 'ARRAY-TOTAL-SIZE-LIMIT) > Symbol: ARRAY-TOTAL-SIZE-LIMIT > Value: 16777216 With OpenMCL I understand the problem - the array is simply too big. But with SBCL it looks like a bug - in 0.8.9 the memory gets trashed for the next GC, not immediately. Also, it seems to be triggered if I try to allocate two smaller arrays - this is how I originally hit the problem. __________________________________ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash |
From: David S. <da...@da...> - 2004-04-24 18:32:14
|
John Klein <jk2...@ya...> writes: > Just to see if I understand where things stand... > > so ARRAY-TOTAL-SIZE-LIMIT is much smaller than the 10^24 elements in my > array. And SBCL shouldn't die when allocating an excessively big > array, I think. $ sbcl This is SBCL 0.8.9.58, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * (> array-total-size-limit (* 4096 4096)) T So SBCL should be OK with the array you are trying to make. > > * (let ((f (make-array '(4096 4096) :initial-element 1d0))) > > (= (aref f 10 11) 99d0)) > > OK, but I _think_ this makes an array of 32 bit pointers pointing > to the same boxed double float. So I think it is half as big as the > 'double-float array. I think you are correct. I'm going to try what you were doing with a MUCH smaller 2d array. I don't want to thrash my machine to death in system calls ;-) * (let ((f (make-array '(10 10) :element-type 'double-float :initial-element 1D0))) (= (aref f 7 8) 99d0)) NIL > With OpenMCL I understand the problem - the array is simply too big. But with > SBCL it looks like a bug - in 0.8.9 the memory gets trashed for the next > GC, not immediately. Also, it seems to be triggered if > I try to allocate two smaller arrays - this is how I originally hit the > problem. I see. Does this work for you? * (loop for i from 1 to 100000 do (let ((f (make-array 4096 :element-type 'double-float))) (= (aref f 10) 1d0))) NIL I'm going to try your original again. * (let ((f (make-array '(4096 4096) :element-type 'double-float))) (= (aref f 10 11) 99d0)) set_auto_gc_trigger: tried to set gc trigger too high! (0x807e37c) fatal error encountered in SBCL pid 1299: lost The system is too badly corrupted or confused to continue at the Lisp level. If the system had been compiled with the SB-LDB feature, we'd drop into the LDB low-level debugger now. But there's no LDB in this build, so we can't really do anything but just exit, sorry. OK, it's not fixed in the latest version yet. This is interesting: * (progn (gc-off) (let ((f (make-array '(4096 4096) :element-type 'double-float))) (= (aref f 10 11) 99d0)) (gc-on) (gc)) fatal error encountered in SBCL pid 1304: GC invariant lost, file "gc-common.c", line 616 The system is too badly corrupted or confused to continue at the Lisp level. If the system had been compiled with the SB-LDB feature, we'd drop into the LDB low-level debugger now. But there's no LDB in this build, so we can't really do anything but just exit, sorry. We get a line number even without LDB enabled. I find this function there: static int scav_other_pointer(lispobj *where, lispobj object) { lispobj first, *first_pointer; gc_assert(is_lisp_pointer(object)); /* Object is a pointer into from space - not FP. */ first_pointer = (lispobj *) native_pointer(object); first = (transother[widetag_of(*first_pointer)])(object); if (first != object) { set_forwarding_pointer(first_pointer, first); #ifdef LISP_FEATURE_GENCGC *where = first; #endif } #ifndef LISP_FEATURE_GENCGC *where = first; #endif gc_assert(is_lisp_pointer(first)); gc_assert(!from_space_p(first)); return 1; } I think it is blowing up on that last gc_assert. For my next build, I'm going to go ahead and enable the LDB. This is an interesting problem. While the code asks a lot from the system, it certainly should be managable. -- I wouldn't mind the rat race so much if it wasn't for all the damn cats. |
From: David S. <da...@da...> - 2004-04-24 21:47:22
|
David Steuber <da...@da...> writes: > For my next build, I'm going to go ahead and enable the LDB. This is > an interesting problem. While the code asks a lot from the system, > it certainly should be managable. Then again, perhaps there is a hard limit. I'm not sure what this means, but this looks like it might be a clue: $ sbcl This is SBCL 0.8.9.58, an implementation of ANSI Common Lisp. More information about SBCL is available at <http://www.sbcl.org/>. SBCL is free software, provided as is, with absolutely no warranty. It is mostly in the public domain; some portions are provided under BSD-style licenses. See the CREDITS and COPYING files in the distribution for more information. * (progn (gc-off) (let ((f (make-array '(4096 4096) :element-type 'double-float))) (= (aref f 10 11) 99d0)) (gc-on) (gc)) set_auto_gc_trigger: tried to set gc trigger too high! (0x8054b54) fatal error encountered in SBCL pid 1948: lost LDB monitor ldb> p 0x8054b54 0x08054b54: odd fixnum: 33641173 ldb> I don't really know how to do anything useful with ldb :-( -- I wouldn't mind the rat race so much if it wasn't for all the damn cats. |
From: John K. <jk2...@ya...> - 2004-04-25 00:58:10
|
--- David Steuber <da...@da...> wrote: > I'm going to try your original again. > > * (let ((f (make-array '(4096 4096) :element-type 'double-float))) > (= (aref f 10 11) 99d0)) > set_auto_gc_trigger: tried to set gc trigger too high! (0x807e37c) > fatal error encountered in SBCL pid 1299: Yes, this is what I get. On Linux with 0.8.5 this slightly more rigorous test (with much less physical memory) works: (let ((f (make-array '(4096 4096) :element-type 'double-float :initial-element 99d0))) (= (aref f 4000 4000) 99d0)) ;; go deep in the array ;; some swap thrashing ==> T (gc :full t) ;; no error at gc __________________________________ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25¢ http://photos.yahoo.com/ph/print_splash |
From: David S. <da...@da...> - 2004-04-25 02:48:51
|
John Klein <jk2...@ya...> writes: > --- David Steuber <da...@da...> wrote: > > > I'm going to try your original again. > > > > * (let ((f (make-array '(4096 4096) :element-type 'double-float))) > > (= (aref f 10 11) 99d0)) > > set_auto_gc_trigger: tried to set gc trigger too high! (0x807e37c) > > fatal error encountered in SBCL pid 1299: > > Yes, this is what I get. > > On Linux with 0.8.5 this slightly more rigorous test (with much less physical > memory) works: I just tested on an old Linux box (384MB RAM) that is in theory far less powerful than my PowerBook G4: $ sbcl This is SBCL 0.8.9.58, an implementation of ANSI Common Lisp. ... * (time (let ((f (make-array '(4096 4096) :element-type 'double-float :initial-element 99d0))) (= (aref f 4000 4000) 99d0))) Evaluation took: 8.791 seconds of real time 6.97394 seconds of user run time 1.427783 seconds of system run time 0 page faults and 134217736 bytes consed. T * (gc :full t) NIL * I've also got sb-thread turned on on this one. It would be nice to get the PPC/Darwin version of SBCL caught up. I think the PPC is a better architecture and should run much faster. I'm still learning Lisp, so I can only contribute so much at the moment. -- I wouldn't mind the rat race so much if it wasn't for all the damn cats. |