Steel Bank Common Lisp

Keith James <kdj@...> writes:

> I recently switched to CL:STABLE-SORT and noticed that my memory use
> was increasing with each operation. Using room and a manual full gc I
> can see that post-sorting the contained simple-arrays are not being
> gc'd as they are in the case of using CL:SORT.
>
> Could this be a bug? I haven't file one on launchpad yet, because I'm
> not sure.

I don't think memory use should increase beyond your maximum vector
size, though that's probably small consolation.  However... from my
reading of the code stable-sorting of vectors uses a temporary vector to
store elements of the vector being sorted, which is not cleared after
use but recycled for the next stable-sort call.  (In this day and age,
I'm not sure why stable-sort / vector-merge-sort doesn't simply create a
new vector for itself, which would at a stroke eliminate both this leaks
and worries about threadsafety -- I certainly can't convince myself that
two threads can't get hold of the same identical
*merge-sort-temp-vector* object.

Cheers,

Christophe

On Thu, Nov 05, 2009 at 12:21:18PM +0000, Christophe Rhodes wrote:

> and worries about threadsafety -- I certainly can't convince myself that
> two threads can't get hold of the same identical
> *merge-sort-temp-vector* object.

make-thread rebinds this non-threadsafe variable so each thread has its
own private version.

  Leslie

>>>>> "Christophe" == Christophe Rhodes <csr21@...> writes:

    Christophe> I don't think memory use should increase beyond your
    Christophe> maximum vector size

Ah, yes. You're correct and I can see that happening now that I'm
using smaller vectors and consequently avoiding exhausting the heap.

Performance is degraded somewhat by using smaller vectors, but it's
not too bad. I'll have a go at patching sbcl to use new vectors as you
suggest.

thanks,

-- 

- Keith James Ph.D. - Wellcome Trust Sanger Institute, UK -


-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

>>>>> "Christophe" == Christophe Rhodes <csr21@...> writes:

    Christophe> In this day and age, I'm not sure
    Christophe> why stable-sort / vector-merge-sort doesn't simply
    Christophe> create a new vector for itself, which would at a
    Christophe> stroke eliminate both this leaks and worries about
    Christophe> threadsafety

I have modified my sbcl by removing *merge-sort-temp-vector* and
replacing it with a fresh vector each time. The result is that memory
use is much more manageable for stable-sorting vectors of big data.
The test suite runs successfully.

I've attached a patch for your consideration.




-- 
 The Wellcome Trust Sanger Institute is operated by Genome Research 
 Limited, a charity registered in England with number 1021457 and a 
 company registered in England with number 2742969, whose registered 
 office is 215 Euston Road, London, NW1 2BE.

On 6 November 2009 00:39, Keith James <kdj@...> wrote:
>>>>>> "Christophe" == Christophe Rhodes <csr21@...> writes:
>
>    Christophe> In this day and age, I'm not sure
>    Christophe> why stable-sort / vector-merge-sort doesn't simply
>    Christophe> create a new vector for itself, which would at a
>    Christophe> stroke eliminate both this leaks and worries about
>    Christophe> threadsafety
>
> I have modified my sbcl by removing *merge-sort-temp-vector* and
> replacing it with a fresh vector each time. The result is that memory
> use is much more manageable for stable-sorting vectors of big data.
> The test suite runs successfully.
>
> I've attached a patch for your consideration.

Thank you! Merged in 1.0.36.18, which also gives the same treatment to
*ZAP-ARRAY-DATA-TEMP*, and truncates the temporary vectors to
zero-length after they are used.

Cheers,

 -- Nikodemus