[Numpy-discussion] mmap (was: fast constructor for arrays from byte data)

[Phil A. <ph...@ge...>]

Chris Barker writes:

> I don't understand mmap at all. From the name, it sounds like the entire
> contents of the file is mapped into memory, so the memory would get used
> as soon as you set it up. If anyone knows, I'd like to hear...

This might be more than you need to know, but there's a 
copy of the article:

"Why aren't you using mmap() yet?"
by Kevin Sheehan at:

http://thrush.eos.ubc.ca/users/phil/mmap/mmap.pdf (note reversed page
order)

and a Solaris/SGI email debate in mhonarc format:

http://thrush.eos.ubc.ca/users/phil/mmap/threads.html

bottom line: you really need a vm implementation that supports
madvise() to get the maximum benefit from mmap.  (Linux 2.4
might have madvise, I haven't tried it out yet -- 2.2 didn't).

The way I used to use mmap was through a numpy array class that was
initiated from a pointer to a mmap'd file.  This made reading/writing
to the file as simple as indexing into the numpy array.  I haven't
been back to reimplement that class on linux since I left solaris,
though.  You can also get some of the benefits of mmap via netcdf,
which will use mmap if it exists.

Regards, Phil