Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Robert K. <ke...@ca...>]

On Tue, Aug 07, 2001 at 03:02:01PM -0700, Chris Barker wrote:
> Robert Kern wrote:
> > On Tue, Aug 07, 2001 at 01:26:57PM -0700, Chris Barker wrote:
> > > I wish there was a fromfile() function.
> > 
> > With Python 2.0 or greater, one can use mmap'ed files as arguments to
> > fromstring.
> 
> Can you give me an example of how to use it? I can not get it to work at
> all, the docs are pretty sketchy. I can't even get a mmap's file to be
> created properly on linux. I havn't tried Windows yet, but I'll need it
> to work there too!

Yeah, it took me some fiddling, too, before I got it to work. The Windows call
to mmap has slightly different semantics in that second argument, but I think 
that if you want to snarf the whole file, then what I do below should work as
well. The Windows API lets you use 0 to default to the whole file, but that's
not portable.

Python 2.0.1 (#0, Jul  3 2001, 12:36:30) 
[GCC 2.95.4 20010629 (Debian prerelease)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import Numeric
>>> import mmap
>>> f = open('earth.dat', 'r+')
>>> f.seek(0, 2)  # seek to EOF
>>> size = f.tell()
>>> f.seek(0)  # rewind; possibly not necessary
>>> m = mmap.mmap(f.fileno(), size)
>>> a = Numeric.fromstring(m)
>>> size
548240
>>> a.shape
(137060,)
>>> size / 4
137060
>>> 

> Also, mmap does not seem to be supported on the Mac, which is where I am
> having memory problems (due to the Mac's lousy memeory management). I'll
> ask about it on the MacPython list.

You're probably SOL, there, but I'll wish you good luck anyways.

if-wishes-were-mmap-implementations-ly yours,

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter