Menu
▾
▴
Re: [Numpy-discussion] Numarray problem on Opteron system, Suse 9.1
|
From: Todd M. <jm...@st...> - 2004-06-11 15:49:28
|
On Fri, 2004-06-11 at 09:37, Jason Ruiter wrote:
> Greetings,
>
Hi,
I think you're getting into bleeding edge territory. For numarray,
the particular problem you exposed should be solved for 1.0, but I
still think the overall 64-bit prognosis is not good. There are API
issues in Python itself which are going to make really large arrays a
problem until they're solved; for instance, the sequence protocol
returns the length of a sequence as an int (32-bits on most platforms).
Regards,
Todd
> I'm running Suse 9.1 on a dual opteron system with 16GB of RAM. I'm
> using Python 2.3.3, Numeric 23.1 and numarray 0.9.
>
> I'm trying to allocate large (>4GB) arrays. Under Numeric, I get:
>
> Python 2.3.3 (#1, Apr 6 2004, 09:45:08)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from Numeric import *
> >>> a=ones(2000000000,Int16);
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "/usr/lib64/python2.3/site-packages/Numeric/Numeric.py", line
> 578, in ones
> a=zeros(shape, typecode, savespace)
> MemoryError: can't allocate memory for array
> >>>
>
> Under numarray:
>
> Python 2.3.3 (#1, Apr 6 2004, 09:45:08)
> [GCC 3.3.3 (SuSE Linux)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
> >>> from numarray import *
> >>> a=ones(2000000000,Int16)
> Traceback (most recent call last):
> File "<stdin>", line 1, in ?
> File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py",
> line 1111, in ones
> retarr = _fillarray(_gen.product(shape), 1, 0, type)
> File "/usr/lib64/python2.3/site-packages/numarray/numarraycore.py",
> line 144, in _fillarray
> outarr = NumArray((size,), outtype)
> ValueError: new_memory: invalid region size: -294967296.
> >>>
>
>
> I've verified that python, Numeric, and numarray are built and linked
> against the 64bit libraries.
>
> I've also verified that it's possible to allocate a >16GB Array with the
> following program I found on one of the debian mailling lists.
> :q:q:q!
> #include <stdio.h>
> #include <stdlib.h>
> int main() { size_t n; void *p; double gb;
> for(gb=20;gb>.3;gb-=.5) {
> n= 1024L * 1024L * 1024L * gb;
> p = malloc( n );
> printf("%12lu %4.1lfGb %p\n",n,n/1024./1024./1024.,p);
> free(p); } return 0; }
>
> 21474836480 20.0Gb (nil)
> 20937965568 19.5Gb (nil)
> 20401094656 19.0Gb (nil)
> 19864223744 18.5Gb (nil)
> 19327352832 18.0Gb (nil)
> 18790481920 17.5Gb (nil)
> 18253611008 17.0Gb (nil)
> 17716740096 16.5Gb 0x3995a02010
> 17179869184 16.0Gb 0x3995a02010
> 16642998272 15.5Gb 0x3995a02010
> 16106127360 15.0Gb 0x3995a02010
> 15569256448 14.5Gb 0x3995a02010
> 15032385536 14.0Gb 0x3995a02010
> 14495514624 13.5Gb 0x3995a02010
> 13958643712 13.0Gb 0x3995a02010
> 13421772800 12.5Gb 0x3995a02010
> 12884901888 12.0Gb 0x3995a02010
> 12348030976 11.5Gb 0x3995a02010
> 11811160064 11.0Gb 0x3995a02010
> 11274289152 10.5Gb 0x3995a02010
> 10737418240 10.0Gb 0x3995a02010
> 10200547328 9.5Gb 0x3995a02010
> 9663676416 9.0Gb 0x3995a02010
> 9126805504 8.5Gb 0x3995a02010
> 8589934592 8.0Gb 0x3995a02010
> 8053063680 7.5Gb 0x3995a02010
> 7516192768 7.0Gb 0x3995a02010
> 6979321856 6.5Gb 0x3995a02010
> 6442450944 6.0Gb 0x3995a02010
> 5905580032 5.5Gb 0x3995a02010
> 5368709120 5.0Gb 0x3995a02010
> 4831838208 4.5Gb 0x3995a02010
> 4294967296 4.0Gb 0x3995a02010
> 3758096384 3.5Gb 0x3995a02010
> 3221225472 3.0Gb 0x3995a02010
> 2684354560 2.5Gb 0x3995a02010
> 2147483648 2.0Gb 0x3995a02010
> 1610612736 1.5Gb 0x3995a02010
> 1073741824 1.0Gb 0x3995a02010
> 536870912 0.5Gb 0x3995a02010
>
>
> Can someone point me in the right direction?
>
> Thanks
> Jason
--
Todd Miller <jm...@st...>
|
