Steel Bank Common Lisp

> Sorry, a bit late for an answer but maybe it still helps...
>
> On Tue, Dec 29, 2009 at 22:23, Nicolas Neuss <neuss@...> wrote:
>
>     Hello,
>    
>     did anybody succeed to make SBCL run on Virtuozzo?
>    
>     I'm stuck with
>    
>      neuss@...:~$ sbcl
>      mmap: Cannot allocate memory
>      ensure_space: failed to validate 536870912 bytes at 0x09000000
>      (hint: Try "ulimit -a"; maybe you should increase memory limits.)
>
> I get that error on my hosteurope VPS (I *think* that uses virtuozzo
> but am not certain).  My woraround is to start sbcl
> with  --dynamic-space-size 200   (depending on the amount of memory
> you are allowed to use, you may have to decrease that number or may be
> able to increase it -- IIRC I used to run happily with 100 until I had
> to load cxml which made sbcl run out of memory in that configuration).

Amazing.  Yes, indeed, this works!  

Now, the only question is if 200 MB is sufficient for my application.
[And it is somewhat sad, because I have 2 GB RAM guaranteed.]

But anyway, thank you very much,

Nicolas

Nicolas Neuss <neuss@...> writes:

> Amazing.  Yes, indeed, this works!  
>
> Now, the only question is if 200 MB is sufficient for my application.
> [And it is somewhat sad, because I have 2 GB RAM guaranteed.]


Update: Also simply calling "sbcl" or increasing the space size does
work for me now.  I suspect that it was an update to Virtuozzo which
made problems go away.

Nicolas

Or your hosting provider noticed the Virtuozzo alerts stemming from you
bumping up against the memory limits and tweaked your configuration.

You can cat /proc/userbeancounters to get the memory stats / limits on your
VPS. If you find that you're still hitting the ceiling on certain
parameters, you may be able to open a support ticket to have them tweak the
settings a bit more.

On Tue, Jan 19, 2010 at 7:22 PM, Nicolas Neuss
<neuss@...:

> Nicolas Neuss <neuss@...> writes:
>
> > Amazing.  Yes, indeed, this works!
> >
> > Now, the only question is if 200 MB is sufficient for my application.
> > [And it is somewhat sad, because I have 2 GB RAM guaranteed.]
>
>
> Update: Also simply calling "sbcl" or increasing the space size does
> work for me now.  I suspect that it was an update to Virtuozzo which
> made problems go away.
>
> Nicolas
>
>
> ------------------------------------------------------------------------------
> Throughout its 18-year history, RSA Conference consistently attracts the
> world's best and brightest in the field, creating opportunities for
> Conference
> attendees to learn about information security's most important issues
> through
> interactions with peers, luminaries and emerging and established companies.
> http://p.sf.net/sfu/rsaconf-dev2dev
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
>

It's very intresting question ( how to work with big memory).
Unfortunatelly I do not know the answer 
Please give here an answer if you will find it. 

Adam

On Sat, Jan 9, 2010 at 12:24 AM, Alistair Gee <alistair.gee@...:

> I am using sbcl 1.0.29 for data analysis on ubuntu 9.10 Linux x86-64
> with 96GB RAM.
>
> sbcl dies at about 8GB of heap usage, regardless of the what I specify
> for --dynamic-space-size on the command line.
>

How does it die?

-- 
Juho Snellman

On Thu, Jan 14, 2010 at 10:28 AM, Juho Snellman <jsnell@...> wrote:
> On Sat, Jan 9, 2010 at 12:24 AM, Alistair Gee <alistair.gee@...>
> wrote:
>>
>> I am using sbcl 1.0.29 for data analysis on ubuntu 9.10 Linux x86-64
>> with 96GB RAM.
>>
>> sbcl dies at about 8GB of heap usage, regardless of the what I specify
>> for --dynamic-space-size on the command line.
>
> How does it die?
> --
> Juho Snellman
>

I don't remember the message, but it said something like "game over"
or "out of heap space" and drops into ldb.

I have since rebuilt sbcl (from CVS) with dynamic-space-end (in
parms.lisp) increased to #x31ffff0000, and it seems to be OK.

On Thu, Jan 14, 2010 at 7:42 PM, Alistair Gee <alistair.gee@...> wrot

> On Thu, Jan 14, 2010 at 10:28 AM, Juho Snellman <jsnell@...> wrote:
> > On Sat, Jan 9, 2010 at 12:24 AM, Alistair Gee <alistair.gee@...>
> > wrote:
> >> sbcl dies at about 8GB of heap usage, regardless of the what I specify
> >> for --dynamic-space-size on the command line.
> >
> > How does it die?
>
> I don't remember the message, but it said something like "game over"
> or "out of heap space" and drops into ldb.
>
> I have since rebuilt sbcl (from CVS) with dynamic-space-end (in
> parms.lisp) increased to #x31ffff0000, and it seems to be OK.
>

Ok, it sounds like you really weren't running with a larger heap then.
--dynamic-space-size should be able to adjust the size upwards though, so
something must've gone wrong there. The main thing I could think of would be
that you weren't passing --dynamic-space-size in at the right position on
the command line (it's a runtime flag, so it must come before any lisp-side
flags).

Not that it matters to you, since you've got it running with a larger
hardcoded size now.

-- 
Juho Snellman

On Thu, 14 Jan 2010, Yue Li wrote:
> Then run my multithreading program again, and still, all the threads
> occupy one of the two cores when running, and the speed up is very
> trivial.
>
> Here did I miss something on configuring the GC? or I am totally out
> of the correct direction?

I think you hit a common problem.  The official SBCL binaries are built 
single-threaded.  To get a multithreaded build, you must build it 
yourself.

This isn't very hard.  Download and extract the SBCL sources somewhere. 
The INSTALL file tells you what to do and how to enable threads (see 
sections 2.1 and 2.2).

I don't think this is a GC issue.

Later,
Daniel

Hi, Daniel

On Fri, Jan 15, 2010 at 1:23 AM, Daniel Herring <dherring@...> wrote:
> On Thu, 14 Jan 2010, Yue Li wrote:
>>
>> Then run my multithreading program again, and still, all the threads
>> occupy one of the two cores when running, and the speed up is very
>> trivial.
>>
>> Here did I miss something on configuring the GC? or I am totally out
>> of the correct direction?
>
> I think you hit a common problem.  The official SBCL binaries are built
> single-threaded.  To get a multithreaded build, you must build it yourself.
>
> This isn't very hard.  Download and extract the SBCL sources somewhere. The
> INSTALL file tells you what to do and how to enable threads (see sections
> 2.1 and 2.2).
>

Thanks for your help! However, the sbcl installed on my machine is
built by myself, and I already enabled sb-thread following the INSTALL
file, I think if I did that incorrectly, I could not invoke
make-thread to spawn threads.

Cheers,

Yue

> I don't think this is a GC issue.
>
> Later,
> Daniel
>

> Thanks for your help! However, the sbcl installed on my machine is
> built by myself, and I already enabled sb-thread following the INSTALL
> file, I think if I did that incorrectly, I could not invoke
> make-thread to spawn threads.
> 
My guess would be that this is an operating system issue. This based on
the facts that sbcl uses pthread on x86 and pthread doesn't have a way
to hint the operating system how to schedule threads.

A possible thing could be that the operating system is reluctant to
migrate threads to other cores until there is subtantial load on them
over time. So if you run your program for several minutes maybe the os
will migrate one of them.

I may be wrong here to, there could be no/little cost at all to have the
threads on different cores and also no/little cost to migrate them.

best regards,
/larry

Larry Valkama wrote:
> My guess would be that this is an operating system issue.

since the OP is on Linux, maybe the "taskset" command could help?

Hi,

Yue Li <xyly781@...> writes:
>> On Thu, 14 Jan 2010, Yue Li wrote:
>>> Here did I miss something on configuring the GC? or I am totally out
>>> of the correct direction?
>>
>> I don't think this is a GC issue.
>
> Thanks for your help! However, the sbcl installed on my machine is
> built by myself, and I already enabled sb-thread following the INSTALL
> file, I think if I did that incorrectly, I could not invoke
> make-thread to spawn threads.

That is true.

Your problem is indeed a GC issue. The garbage collector is single
threaded and has to stop the world for collection, so everything gets
interrupted during collection. The less you cons, of course, the less
that happens, but if you have something consing like mad, then you will
mostly see the GC thread.

The same thing bit me a few days ago.

The short term solution is patience, or perhaps having multiple sbcls
communicating via sockets, as no other (free) multithreaded CL i am
aware of is competitive with SBCL in terms of speed anyway (at least on
numerical computations). Except perhaps CMU CL, and I doubt that there
things are better on this regard..

The long term solution would be a good parallel GC, but I suspect that
these are not very easy to write.

Regards,
        Mario.

---------- Forwarded message ----------
From: Yue Li <xyly781@...>
Date: Fri, Jan 15, 2010 at 11:49 AM
Subject: Re: [Sbcl-help] SBCL threads only occupy one of the cores
To: Giovanni Gigante <giov@...>


On Fri, Jan 15, 2010 at 6:03 AM, Giovanni Gigante <giov@...> wrote:
> Larry Valkama wrote:
>> My guess would be that this is an operating system issue.
>
> since the OP is on Linux, maybe the "taskset" command could help?
>
  Maybe I was wrong, but I think taskset could be used to put a
process to another core, but could not schedule a thread to occupy
another core?

Yue

On Fri, Jan 15, 2010 at 5:51 AM, Larry Valkama
<remlali@...> wrote:
>
>> Thanks for your help! However, the sbcl installed on my machine is
>> built by myself, and I already enabled sb-thread following the INSTALL
>> file, I think if I did that incorrectly, I could not invoke
>> make-thread to spawn threads.
>>
> My guess would be that this is an operating system issue. This based on
> the facts that sbcl uses pthread on x86 and pthread doesn't have a way
> to hint the operating system how to schedule threads.
>
> A possible thing could be that the operating system is reluctant to
> migrate threads to other cores until there is subtantial load on them
> over time. So if you run your program for several minutes maybe the os
> will migrate one of them.
>

Though I'm still not sure the reason is due to OS scheduler, but the
behavior you just described above is quite true. If I ask each thread
to execute a time cost load, say a deadloop, or then, threads will
also occupy the other core. But the strange thing is that, after
executing for a while, the threads will come back to the first core
from the second one again.

Thanks,

Yue


> I may be wrong here to, there could be no/little cost at all to have the
> threads on different cores and also no/little cost to migrate them.
>
> best regards,
> /larry
>
> ------------------------------------------------------------------------------
> Throughout its 18-year history, RSA Conference consistently attracts the
> world's best and brightest in the field, creating opportunities for Conference
> attendees to learn about information security's most important issues through
> interactions with peers, luminaries and emerging and established companies.
> http://p.sf.net/sfu/rsaconf-dev2dev
> _______________________________________________
> Sbcl-help mailing list
> Sbcl-help@...
> https://lists.sourceforge.net/lists/listinfo/sbcl-help
>

In article 
<3df685d31001141817s333c8bf7l209dd80a953942fa@...>,
 Yue Li <xyly781@...> wrote:

> I'm running some simple computation, such as computing the sum of a
> list of one million integers in a reduction fashion.  I would like to
> parallelize the sum by using threads.  Therefore, I usually create
> about 2-4 threads, each computes a portion of the sum. The results are
> always correct, however, I found the performance is always as fast as
> my sequential version. Then I looked at the CPU monitor, and it shows
> that when doing computation using threads, all the threads only occupy
> the first core, and very seldomly occupy the second core. I read the
> SBCL manual, and googled on websites but seems I could not find the
> answer.

Can you post or link to (a reduced version of) your test case?

Paul Khuong

In article <hi7kgf$4rh$1@...>,
 Tamas K Papp <tkpapp@...> wrote:

> I have a simple array of a numerical type (float or integer), and I
> need to convert it to another type (eg with coerce).
> 
> What's the fastest way to do this in SBCL?  I tried

You want MAP-INTO, e.g. 
  (map-into (make-array (length vector)
                        :element-type '(complex double-float))
            #'complex
            vector)

MAP should probably be made smarter on such cases, granted. You've 
noticed that COERCE isn't smart when coercing to complexes; you'll have 
to drive the conversion explicitly.

Paul Khuong