Re: [Sbcl-devel] GSoC 2013 projects - priorities?

[Nikodemus S. <nik...@ra...>]

On 23 April 2013 01:56, Paul Khuong <pv...@pv...> wrote:

>> I'll be starting a CS PhD program in the fall, working on bioinformatics
>> problems where current C++ software uses in excess of .5 TB of RAM.  I'd
>> like to be able to use SBCL for any work where C/++ isn't truly required,
>> so my biggest motivation is space efficiency.
>
> Unless that's mostly unboxed data, I don't think your biggest issue will
> be conservativeness as much as the fact that our GC is mostly copying.
> If it is mostly unboxed data, conservativeness may very well not be an
> issue at all; the vast majority of programs work fine, after all. I also
> wouldn't be surprised if you end up doing something like ITA (now
> google): they use SB-ALIEN to represent huge data on the C heap. Perhaps
> you want to try to come up with a project that would directly help your
> use case?

A quick suggestion: way to point an array-header into an mmap'ed file
or memory containing unboxed data.

Current array-headers (used for non-simple arrays) always point to
simple-arrays which provide the actual storage; for large datasets
having one pointing to a mmap'd file instead would seem like a boon.

(There are at least a couple libraries floating around that do
something similar by sticking a widetag and length into foreign
memory, but that's not ideal as it means you cannot examine arbitrary
foreign memory as a lisp array, but only foreign memory where two
first words are free for you to write into.)

Cheers,

 -- nikodemus