From: Jacek P. <rub...@go...> - 2014-02-02 19:06:13
|
This is my first significant Lisp Program. And I still consider myself a noob. When I run tests I run out of heap. This is my test file: https://github.com/bigos/discompiler/blob/master/test/lisp-unit.lisp Somebody has suggested to ask this question here instead of comp.lang.lisp Does anybody have an idea how should I troubleshoot that? |
From: Christophe R. <cs...@ca...> - 2014-02-02 20:38:24
|
Jacek Podkanski <rub...@go...> writes: > This is my first significant Lisp Program. And I still consider myself a > noob. When I run tests I run out of heap. This is my test file: > https://github.com/bigos/discompiler/blob/master/test/lisp-unit.lisp > > Somebody has suggested to ask this question here instead of comp.lang.lisp > Does anybody have an idea how should I troubleshoot that? Running out of heap is what happens when you retain references to data. So, one thing you could do is find out whether any of your operators are causing the retention of data unnecessarily; you could investigate by running explicit garbage collections (using SB-EXT:GC) and then querying the heap using ROOM. This might make the problem go away. If it does, you might be a victim of conservatism in the garbage collector; under those circumstances, keeping the explicit garbage collections in is not wrong. If it doesn't, then there might be something in your code which retains data unnecessarily; finding it and stopping it from doing that is the next thing to do. To identify individual functions responsible for the allocation of data, you can use sb-sprof in allocation profile mode, or the deterministic profiler; these will tell you what is responsible for allocation, but not necessarily what's causing the retention of data. Good luck, Christophe |
From: Jacek P. <rub...@go...> - 2014-02-02 23:54:18
|
I have been digging through the code whole evening. I got rid of 2 evals and moved md5 sum testing to a separate test. In my profile there's still one suspicious function that I need to check after the break. Number of samples: 1000 Alloc interval: 4 regions (approximately 128 kB) Total sampling amount: 4000 regions (approximately 128000 kB) Number of cycles: 0 Sampled threads: #<SB-THREAD:THREAD "repl-thread" RUNNING {10041A0063}> Self Total Cumul Nr Count % Count % Count % Calls Function ------------------------------------------------------------------------ -------------- cut --------------- 30 2 0.2 553 55.3 943 94.3 - IMPORTED-FUNCTION-NAMES Dynamic space usage is normally around 560,000,000 bytes, but on one occasion when running tests I got this: Dynamic space usage is: 883,161,536 bytes. Read-only space usage is: 5,952 bytes. Static space usage is: 4,000 bytes. Control stack usage is: 9,984 bytes. Binding stack usage is: 1,104 bytes. Control and binding stack usage is for the current thread only. Garbage collection is currently enabled. The input data does not change, but when I got run out of heap error I had above profiler report. Did I run the test one after another too soon not giving enough time for garbage collector to complete it's job? On 02/02/14 20:38, Christophe Rhodes wrote: > Jacek Podkanski <rub...@go...> writes: > >> This is my first significant Lisp Program. And I still consider myself a >> noob. When I run tests I run out of heap. This is my test file: >> https://github.com/bigos/discompiler/blob/master/test/lisp-unit.lisp >> >> Somebody has suggested to ask this question here instead of comp.lang.lisp >> Does anybody have an idea how should I troubleshoot that? > Running out of heap is what happens when you retain references to data. > So, one thing you could do is find out whether any of your operators are > causing the retention of data unnecessarily; you could investigate by > running explicit garbage collections (using SB-EXT:GC) and then querying > the heap using ROOM. > > This might make the problem go away. If it does, you might be a victim > of conservatism in the garbage collector; under those circumstances, > keeping the explicit garbage collections in is not wrong. If it > doesn't, then there might be something in your code which retains data > unnecessarily; finding it and stopping it from doing that is the next > thing to do. > > To identify individual functions responsible for the allocation of data, > you can use sb-sprof in allocation profile mode, or the deterministic > profiler; these will tell you what is responsible for allocation, but > not necessarily what's causing the retention of data. > > Good luck, > > Christophe > |
From: Jacek P. <rub...@go...> - 2014-02-03 00:01:27
|
This is the function that lies hight up in profiler report (defun imported-function-names (mem bytes imp-dir-rva) (loop for il from imp-dir-rva by 4 for ilx = (bytes-to-type-int (get-allocated-bytes mem (rva-addr il bytes) 4)) until (zerop ilx) collect (list il ilx (if (import-by-ordinalp bytes ilx) (ldb (byte 16 0) ilx) (cons (imported-function-hint mem bytes ilx) (imported-function-name mem bytes ilx)))))) Could hundreds of conses cause such garbage collection problem? On 02/02/14 20:38, Christophe Rhodes wrote: > Jacek Podkanski <rub...@go...> writes: > >> This is my first significant Lisp Program. And I still consider myself a >> noob. When I run tests I run out of heap. This is my test file: >> https://github.com/bigos/discompiler/blob/master/test/lisp-unit.lisp >> >> Somebody has suggested to ask this question here instead of comp.lang.lisp >> Does anybody have an idea how should I troubleshoot that? > Running out of heap is what happens when you retain references to data. > So, one thing you could do is find out whether any of your operators are > causing the retention of data unnecessarily; you could investigate by > running explicit garbage collections (using SB-EXT:GC) and then querying > the heap using ROOM. > > This might make the problem go away. If it does, you might be a victim > of conservatism in the garbage collector; under those circumstances, > keeping the explicit garbage collections in is not wrong. If it > doesn't, then there might be something in your code which retains data > unnecessarily; finding it and stopping it from doing that is the next > thing to do. > > To identify individual functions responsible for the allocation of data, > you can use sb-sprof in allocation profile mode, or the deterministic > profiler; these will tell you what is responsible for allocation, but > not necessarily what's causing the retention of data. > > Good luck, > > Christophe > |
From: Jacek P. <rub...@go...> - 2014-02-03 01:04:33
|
Adding (sb-ext:gc :full T) at the end of every problematic test seems to fix the problem. On 02/02/14 20:38, Christophe Rhodes wrote: > Jacek Podkanski <rub...@go...> writes: > >> This is my first significant Lisp Program. And I still consider myself a >> noob. When I run tests I run out of heap. This is my test file: >> https://github.com/bigos/discompiler/blob/master/test/lisp-unit.lisp >> >> Somebody has suggested to ask this question here instead of comp.lang.lisp >> Does anybody have an idea how should I troubleshoot that? > Running out of heap is what happens when you retain references to data. > So, one thing you could do is find out whether any of your operators are > causing the retention of data unnecessarily; you could investigate by > running explicit garbage collections (using SB-EXT:GC) and then querying > the heap using ROOM. > > This might make the problem go away. If it does, you might be a victim > of conservatism in the garbage collector; under those circumstances, > keeping the explicit garbage collections in is not wrong. If it > doesn't, then there might be something in your code which retains data > unnecessarily; finding it and stopping it from doing that is the next > thing to do. > > To identify individual functions responsible for the allocation of data, > you can use sb-sprof in allocation profile mode, or the deterministic > profiler; these will tell you what is responsible for allocation, but > not necessarily what's causing the retention of data. > > Good luck, > > Christophe > |
From: Jacek P. <rub...@go...> - 2014-02-03 23:06:16
|
Adding this code at the end of my tests bring heap use back to normal. (setf bytes nil mem nil) (sb-ext:gc :full T) |
From: Hugo C. S. <hug...@gm...> - 2014-02-04 19:05:10
|
Though I'm an absolute beginner in CL I took a look to the code, I just missed a bit more of information for describing the problem, and also I couldn't really run the code since unit tests use some files not in the repo (currently myfavlibrary.exe). I think the main problem is the file-to-bytes function, you are loading the entire file in memory (though I don't know the size of that specific file which made the test crash). Also the function bytes which splices the big vector into a list of bytes. As I said I'm a beginner, maybe all that is not correct, I would use arrays instead of lists in the bytes function since you always pass the size (in count param). Maybe you dont need at all copying splices of the big array into new objects since they get unreachable soon (they got translated to ints, longs, or so). As for avoiding load the entire file in mem I don't know a nice solution, you can always use file-position to seek in the file but I don't know if doing this the reads get buffered. Also I noted you use loop with index for arrays, you can use across for sequences and subseq for splicing the sequences. I'm calling splice to a subsequence sorry if it sounds misleading. A simple test over a (make-array 10000000 :element-type 'integer) performed 20% faster with across than with the index approach summing all zeros in the array. Oh! also a tip, (loop for i to (1- x) == (loop for i below x, well maybe you tip more this way... 2014-02-04 Jacek Podkanski <rub...@go...>: > Adding this code at the end of my tests bring heap use back to normal. > > (setf bytes nil > mem nil) > (sb-ext:gc :full T) > > > ------------------------------------------------------------------------------ > Managing the Performance of Cloud-Based Applications > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > Read the Whitepaper. > > http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > _______________________________________________ > Sbcl-help mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-help > |
From: Sanel Z. <sa...@gm...> - 2014-02-04 15:16:24
|
I'm curious, shouldn't GC be automatically called when heap near limit was reached? Best, Sanel On Tue, Feb 4, 2014 at 12:06 AM, Jacek Podkanski <rub...@go...> wrote: > Adding this code at the end of my tests bring heap use back to normal. > > (setf bytes nil > mem nil) > (sb-ext:gc :full T) > > ------------------------------------------------------------------------------ > Managing the Performance of Cloud-Based Applications > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > Read the Whitepaper. > http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > _______________________________________________ > Sbcl-help mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-help |
From: Jacek P. <rub...@go...> - 2014-02-04 15:31:52
|
For some reason it doesn't happen on my system. With explict GC I can run tests and watch Youtube on a 1GB machine. If I don't use explicit GC I run out of memory. On 04/02/14 15:15, Sanel Zukan wrote: > I'm curious, shouldn't GC be automatically called when heap near limit > was reached? > > Best, > Sanel > > On Tue, Feb 4, 2014 at 12:06 AM, Jacek Podkanski > <rub...@go...> wrote: >> Adding this code at the end of my tests bring heap use back to normal. >> >> (setf bytes nil >> mem nil) >> (sb-ext:gc :full T) >> >> ------------------------------------------------------------------------------ >> Managing the Performance of Cloud-Based Applications >> Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. >> Read the Whitepaper. >> http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk >> _______________________________________________ >> Sbcl-help mailing list >> Sbc...@li... >> https://lists.sourceforge.net/lists/listinfo/sbcl-help > ------------------------------------------------------------------------------ > Managing the Performance of Cloud-Based Applications > Take advantage of what the Cloud has to offer - Avoid Common Pitfalls. > Read the Whitepaper. > http://pubads.g.doubleclick.net/gampad/clk?id=121051231&iu=/4140/ostg.clktrk > _______________________________________________ > Sbcl-help mailing list > Sbc...@li... > https://lists.sourceforge.net/lists/listinfo/sbcl-help > |
From: Stig H. <sti...@gm...> - 2014-02-04 15:34:13
|
On Tue, Feb 4, 2014 at 4:15 PM, Sanel Zukan <sa...@gm...> wrote: > I'm curious, shouldn't GC be automatically called when heap near limit > was reached? > It definitely should. However, calling it manually is harmless except for wasting CPU time. Calling first (GC :full t) and then (ROOM) is a test to find out if you have generated uncollectable garbage, that is objects you don't need but still have a reference to somewhere. |
From: Christophe R. <cs...@ca...> - 2014-02-04 15:56:06
|
Sanel Zukan <sa...@gm...> writes: > I'm curious, shouldn't GC be automatically called when heap near limit > was reached? Yes, but: the garbage collector strategy is a "mostly copying" strategy: data that is still live needs to be copied to a new location rather than remaining in place. One problem that this can cause is that if there is not much reclaimable space, the copying as part of garbage collection itself can cause the heap to run out of space. Cheers, Christophe |