From: Karol L. <kar...@kn...> - 2006-07-25 06:59:41
|
On Tuesday 25 July 2006 02:36, Mike Ressler wrote: > I'm trying to work with memmaps on very large files, i.e. > 2 GB, up to 10 > GB. The files are data cubes of images (my largest is > 1290(x)x1024(y)x2011(z)) and my immediate task is to strip the data from > 32-bits down to 16, and to rearrange some of the data on a per-xy-plane > basis. I'm running this on a Fedora Core 5 64-bit system, with > python-2.5b2(that I believe I compiled in 64-bit mode) and > numpy-1.0b1. The disk has 324 GB free space. > > The log from a minimal case is as follows: > > ressler > python2.5 > Python 2.5b2 (r25b2:50512, Jul 18 2006, 12:58:29) > [GCC 4.1.1 20060525 (Red Hat 4.1.1-1)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > > >>> import numpy as np > >>> data=np.memmap('temp_file',mode='w+',shape=(2011,1280,1032),dtype='h') > > size = 2656450560 > bytes = 5312901120 > len(mm) = 5312901120 > (2011, 1280, 1032) h 0 0 > Traceback (most recent call last): > File "<stdin>", line 1, in <module> > File "/usr/local/lib/python2.5/site-packages/numpy/core/memmap.py", line > 75, in __new__ > offset=offset, order=order) > TypeError: buffer is too small for requested array > > > If I have a small number of frames (z=800 rather than 2011), this all works > fine. I've added a few lines to memmap.py to print some diagnostic > information - the error occurs on line 71 in the original memmap.py file, > not 75. The "size =" and "bytes =" lines show that memmap.py is calculating > the correct size for the buffer, and the len(mm) shows that the python > mmap.mmap call on line 67 is returning a buffer of the correct size. The > "(2011, 1280, 1032) h 0 0" bit is from a print statement that was left in > the source file by the authors, and indicates what the following "self = > ndarray.__new__" call is trying to do. However, it is the ndarray.__new__ > call that is breaking down, and I don't really have enough skill to > continue chasing it down. I took a quick look at the C source, but I > couldn't figure out where the ndarray.__new__ is actually defined. > > Any suggestions to help me past this? Thanks. > > Mike I know Travis has nswered in a different thread. Let me jsut add where the actual error is raised - maybe it will be of some use. It is around line 5490 of arrayobject.c (procedure array_new): else { /* buffer given -- use it */ if (dims.len == 1 && dims.ptr[0] == -1) { dims.ptr[0] = (buffer.len-(intp)offset) / itemsize; } else if ((strides.ptr == NULL) && \ buffer.len < itemsize* \ PyArray_MultiplyList(dims.ptr, dims.len)) { PyErr_SetString(PyExc_TypeError, "buffer is too small for " \ "requested array"); goto fail; } So it does look like an overflow to me. Karol -- written by Karol Langner wto lip 25 08:56:42 CEST 2006 |