numpy-discussion — Discussion list for all users of Numerical Python

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Chris B. <chr...@ho...>]

Robert Kern wrote:

> > Thanks, that works, but now I am wondering: what I want is a fast and
> > memory efficient way to get the contents of a file into a NujmPy array,
> > this sure doesn't look any better than:
> >
> > a = fromstring(file.read()))
> 
> Depends on how large the file is. file.read() creates a temporary string the
> size of the file. That string isn't freed until fromstring() finishes and
> returns the array object. For a brief time, both the string and the array have
> duplicates of the same data taking up space in memory.

Exactly. Also is the memory used for hte string guarranteed to be freed
right away? I have no idea how Python internals work.

Anyway, that's why I want a "fromfile()" method, like the one inthe
library array module.
 
> I don't know the details of mmap, so it's certainly possible that the only way
> that fromstring knows how to access the data is to pull all of it into memory
> first, thus recreating the problem. Alas.

I don't understand mmap at all. From the name, it sounds like the entire
contents of the file is mapped into memory, so the memory would get used
as soon as you set it up. If anyone knows, I'd like to hear...


-Chris





-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] swig and Numeric

[Scott R. <ra...@cf...>]

"Paul F. Dubois" wrote:
> 
> Does someone have an example of using SWIG to connect Numeric arrays to C code
> that expects double* arguments?

Hi Paul,

I have a fairly comprehensive set of typedefs (including 1-D and 2-D
array input and output) for interfacing Numeric arrays here:

ftp://cfa-ftp.harvard.edu/pub/ransom/numpy.i

As an example, say you have a function with a prototype:

int foo(double *array, int len_array);

You would simply make a ".i" file with the following:

======================
%module foo
%include numpy.i 

%apply double* IN_1D_DOUBLE { double *array };
int bar(double *array, int len_array);

=======================

Run swig on it, compile it, and import it.

Hope this helps,

Scott

-- 
Scott M. Ransom                   Address:  Harvard-Smithsonian CfA
Phone:  (617) 496-7908                      60 Garden St.  MS 10 
email:  ra...@cf...              Cambridge, MA  02138
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

Re: [Numpy-discussion] permutation and array subsets

[Scott R. <ra...@cf...>]

Serge Rey wrote:
>
> i'm trying to use the results of permutation (from RandomArray.py) to
> reorder the rows of an array:
> 
> >>> r=arange(20)
> >>> id=permutation(20)
> >>> y=r[id]

Try:

y = take(r, permutation(len(r)))

Scott

-- 
Scott M. Ransom                   Address:  Harvard-Smithsonian CfA
Phone:  (617) 496-7908                      60 Garden St.  MS 10 
email:  ra...@cf...              Cambridge, MA  02138
GPG Fingerprint: 06A9 9553 78BE 16DB 407B  FFCA 9BFA B6FF FFD3 2989

[Numpy-discussion] permutation and array subsets

[Serge R. <se...@ro...>]

i'm trying to use the results of permutation (from RandomArray.py) to
reorder the rows of an array:

>>> r=arange(20)
>>> id=permutation(20)
>>> y=r[id]
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  IndexError: invalid index
  >>> 

i can use a for loop to permutate the ordering of r, but that has a
speed hit associated with it.  can anyone offer a suggestion for a way
to do this more efficiently? 

thanks,
serge
-- 
Sergio J. Rey	http://typhoon.sdsu.edu/rey.html

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Robert K. <ke...@ca...>]

On Tue, Aug 07, 2001 at 04:33:30PM -0700, Chris Barker wrote:

[snip]

> Thanks, that works, but now I am wondering: what I want is a fast and
> memory efficient way to get the contents of a fjile into a NujmPy array,
> this sure doesn't look any better than:
> 
> a = fromstring(file.read()))

Depends on how large the file is. file.read() creates a temporary string the
size of the file. That string isn't freed until fromstring() finishes and
returns the array object. For a brief time, both the string and the array have
duplicates of the same data taking up space in memory.

I don't know the details of mmap, so it's certainly possible that the only way
that fromstring knows how to access the data is to pull all of it into memory 
first, thus recreating the problem. Alas.

> thanks anyway,
> 
> -Chris

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Chris B. <chr...@ho...>]

Robert Kern wrote:

> Yeah, it took me some fiddling, too, before I got it to work. The Windows call
> to mmap has slightly different semantics in that second argument, but I think
> that if you want to snarf the whole file, then what I do below should work as
> well. The Windows API lets you use 0 to default to the whole file, but that's
> not portable.

[code snipped]

no, the "0 argument does not seem to work on Unix. It seems odd to not
have a way to just snarf the whole file automatically, but so be it

Thanks, that works, but now I am wondering: what I want is a fast and
memory efficient way to get the contents of a fjile into a NujmPy array,
this sure doesn't look any better than:

a = fromstring(file.read()))

thanks anyway,

-Chris


-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Robert K. <ke...@ca...>]

On Tue, Aug 07, 2001 at 03:02:01PM -0700, Chris Barker wrote:
> Robert Kern wrote:
> > On Tue, Aug 07, 2001 at 01:26:57PM -0700, Chris Barker wrote:
> > > I wish there was a fromfile() function.
> > 
> > With Python 2.0 or greater, one can use mmap'ed files as arguments to
> > fromstring.
> 
> Can you give me an example of how to use it? I can not get it to work at
> all, the docs are pretty sketchy. I can't even get a mmap's file to be
> created properly on linux. I havn't tried Windows yet, but I'll need it
> to work there too!

Yeah, it took me some fiddling, too, before I got it to work. The Windows call
to mmap has slightly different semantics in that second argument, but I think 
that if you want to snarf the whole file, then what I do below should work as
well. The Windows API lets you use 0 to default to the whole file, but that's
not portable.

Python 2.0.1 (#0, Jul  3 2001, 12:36:30) 
[GCC 2.95.4 20010629 (Debian prerelease)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> import Numeric
>>> import mmap
>>> f = open('earth.dat', 'r+')
>>> f.seek(0, 2)  # seek to EOF
>>> size = f.tell()
>>> f.seek(0)  # rewind; possibly not necessary
>>> m = mmap.mmap(f.fileno(), size)
>>> a = Numeric.fromstring(m)
>>> size
548240
>>> a.shape
(137060,)
>>> size / 4
137060
>>> 

> Also, mmap does not seem to be supported on the Mac, which is where I am
> having memory problems (due to the Mac's lousy memeory management). I'll
> ask about it on the MacPython list.

You're probably SOL, there, but I'll wish you good luck anyways.

if-wishes-were-mmap-implementations-ly yours,

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Chris B. <chr...@ho...>]

Robert Kern wrote:
> On Tue, Aug 07, 2001 at 01:26:57PM -0700, Chris Barker wrote:
> > I wish there was a fromfile() function.
> 
> With Python 2.0 or greater, one can use mmap'ed files as arguments to
> fromstring.

Can you give me an example of how to use it? I can not get it to work at
all, the docs are pretty sketchy. I can't even get a mmap's file to be
created properly on linux. I havn't tried Windows yet, but I'll need it
to work there too!

Also, mmap does not seem to be supported on the Mac, which is where I am
having memory problems (due to the Mac's lousy memeory management). I'll
ask about it on the MacPython list.

-Chris



-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Robert K. <ke...@ca...>]

On Tue, Aug 07, 2001 at 01:26:57PM -0700, Chris Barker wrote:

[snip]

> Note that you end up creating a Python string of the data first, and
> then a NumPy array from that. This doesn't really cost that much, but it
> can be an issue with huge data sets. I wish there was a fromfile()
> function. I may get around to writing it one day.

With Python 2.0 or greater, one can use mmap'ed files as arguments to
fromstring.

> -Chris

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

RE: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Paul F. D. <pa...@pf...>]

See fromstring() 
>>> import Numeric
>>> x=Numeric.arange(10)
>>> s=x.tostring()
>>> y = Numeric.fromstring(s)
>>> y
array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])
>>> 

-----Original Message-----
From: num...@li...
[mailto:num...@li...]On Behalf Of Hung
Jung Lu
Sent: Tuesday, August 07, 2001 12:50 PM
To: num...@li...
Subject: [Numpy-discussion] fast constructor for arrays from byte data
(unpickling?)


Hi,

Arrays have a method called tostring() which generates
the binary data. Is there an inverse function to that?
That is, generating an array from the binary data
string?

For large matrices, pickling/unpickling is a bit too
much overhead (data stored as ASCII character strings
instead of binary data strings.) I know, I am talking
about a factor 4 here. But there is a big difference
between 1 minute loading time and 4 minute loading
time.

I would imagine this is a very common problem/request
for people working with large matrices. And I am sure
hacking the C code to provide another fast constructor
for arrays from binary strings won't be too hard. The
questions is: has anyone already tried it? Is it
already there?

(For the kludge masters: one kludge is of course to
store the binary data on disk, then use cStringIO to
build the pickled file and then unpickle from the
cStringIO. Speed is probably OK since the pickled file
lives on RAM. But that's a kludge. :) )

regards,

Hung Jung


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

_______________________________________________
Numpy-discussion mailing list
Num...@li...
http://lists.sourceforge.net/lists/listinfo/numpy-discussion

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Hung J. Lu <hun...@ya...>]

> > That is, generating an array from the binary data
> > string?
> 
> Numeric.fromstring

Ahhh! Now that seems so obvious. :) (Banging my head.)

thanks!

Hung Jung


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Chris B. <chr...@ho...>]

Hung Jung Lu wrote:
> Arrays have a method called tostring() which generates
> the binary data. Is there an inverse function to that?
> That is, generating an array from the binary data
> string?

As you suspected, there is such a thing, and it is called (surprise):
fromstring(). It is a function, rather than a method, as it is a
constructor for an array. To get data from a file, you have to get it
into a string first, so I use:

M = fromstring(file.read(),Float)

This makes a rank -1 array. You might have to reshape it.

Note that you end up creating a Python string of the data first, and
then a NumPy array from that. This doesn't really cost that much, but it
can be an issue with huge data sets. I wish there was a fromfile()
function. I may get around to writing it one day.

-Chris







-- 
Christopher Barker,
Ph.D.                                                           
Chr...@ho...                 ---           ---           ---
http://members.home.net/barkerlohmann ---@@       -----@@       -----@@
                                   ------@@@     ------@@@     ------@@@
Oil Spill Modeling                ------   @    ------   @   ------   @
Water Resources Engineering       -------      ---------     --------    
Coastal and Fluvial Hydrodynamics --------------------------------------
------------------------------------------------------------------------

Re: [Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Robert K. <ke...@ca...>]

On Tue, Aug 07, 2001 at 12:50:04PM -0700, Hung Jung Lu wrote:
> Hi,
> 
> Arrays have a method called tostring() which generates
> the binary data. Is there an inverse function to that?
> That is, generating an array from the binary data
> string?

Numeric.fromstring

HTH.

-- 
Robert Kern
ke...@ca...

"In the fields of hell where the grass grows high
 Are the graves of dreams allowed to die."
  -- Richard Harter

[Numpy-discussion] fast constructor for arrays from byte data (unpickling?)

[Hung J. Lu <hun...@ya...>]

Hi,

Arrays have a method called tostring() which generates
the binary data. Is there an inverse function to that?
That is, generating an array from the binary data
string?

For large matrices, pickling/unpickling is a bit too
much overhead (data stored as ASCII character strings
instead of binary data strings.) I know, I am talking
about a factor 4 here. But there is a big difference
between 1 minute loading time and 4 minute loading
time.

I would imagine this is a very common problem/request
for people working with large matrices. And I am sure
hacking the C code to provide another fast constructor
for arrays from binary strings won't be too hard. The
questions is: has anyone already tried it? Is it
already there?

(For the kludge masters: one kludge is of course to
store the binary data on disk, then use cStringIO to
build the pickled file and then unpickle from the
cStringIO. Speed is probably OK since the pickled file
lives on RAM. But that's a kludge. :) )

regards,

Hung Jung


__________________________________________________
Do You Yahoo!?
Make international calls for as low as $.04/minute with Yahoo! Messenger
http://phonecard.yahoo.com/

[Numpy-discussion] swig and Numeric

[Paul F. D. <du...@ll...>]

Does someone have an example of using SWIG to connect Numeric arrays to C=
 code
that expects double* arguments?

[Numpy-discussion] NumTut

[<gb...@ma...>]

RGVhciB3aG8gbWF5IGJlIGNvbmNlcm5lZCwNCg0KSSB3YW50IHRvIGluc3RhbGwgdGhlIHBhY2th
Z2UgTnVtVHV0LiBCdXQgSSBjYW4ndCBmaW5kIHRoZSBkaXJlY3Rvcnkgb2Ygc291cmNlIGRpc3Ry
aWJ1dGlvbi4gUGxlYXNlIGxldCBtZSBrbm93IGhvdyB0byBnZXQgdGhlIGRpcmVjdG9yeSBvZGYg
c291Y2UgZGlzdHJpYnV0aW9uIGFuZCB0aGUgc3ViZGlyZWN0b3J5IERlbW8uDQoNClRoYW5rcw0K
DQpHeW95LUJvbmcgTGVlIGZyb20gS29yZWENCg==

[Numpy-discussion] new to list

[Rob <eu...@ho...>]

Hello,

I'm new to the list so I thought I'd introduce myself.  I use Numpy at
work for some Python programs that interface with Agilent and/or Anritsu
microwave equipment, using the GPIB bus.

However, my main interest lately is porting some EM codes into Numeric
Python.  I've had varied success.  See my web site
www.members.home.net/europax.  Right now I'm trying to Numpy-ize this
FEM-MOM code.  I've profiled it using the Python profile class.  Using
MOM for a dipole takes 5 minutes of cpu :) :)  Interestingly, some of
the worst routines are simple ones that really have little to do with
matrix math.
Anyway, I'm having fun.  Its wonderful to be able to focus on algorithms
rather than wading through arcane programming syntax !!

Rob.

Re: [Numpy-discussion] A savespace bug, perhaps?

[Travis O. <oli...@ie...>]

On Fri, 03 Aug 2001, Nathaniel Gray wrote:
> Hi everybody,
>
> I'm using Python 2.1.1 and Numpy 20.1.0  This seems to be a problem:
>
> Python 2.1.1 (#2, Jul 31 2001, 14:10:42)
> [GCC 2.96 20000731 (Linux-Mandrake 8.0 2.96-0.48mdk)] on linux2
> Type "copyright", "credits" or "license" for more information.
>
> >>> from Numeric import *
> >>> from MLab import zeros
> >>> joe = zeros(3,'l',1)
> >>> fred = array([1,2,3],'l')
> >>> joe += fred
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
> TypeError: return array has incorrect type
>
> This fails whether or not fred has the spacesaver flag set.
>
> This, however, works fine:
> >>> joe = zeros(3, 'l')
> >>> fred = array([1,2,3], 'l')
> >>> joe += fred
> >>> joe
>
> array([1, 2, 3])
>
> Am I doing something wrong or is this a bug?
>
> Please CC to me since I'm not subscribed to the list.
> Thanks a lot!
> -Nathan

Thanks for the bug report, Nathan. 

It was indeed a bug, which I hope has now been squashed in the latest CVS 
version of Numeric.  

Thanks for your help.

Travis

[Numpy-discussion] A savespace bug, perhaps?

[Nathaniel G. <n8...@ca...>]

Hi everybody,

I'm using Python 2.1.1 and Numpy 20.1.0  This seems to be a problem:

Python 2.1.1 (#2, Jul 31 2001, 14:10:42)                                     
[GCC 2.96 20000731 (Linux-Mandrake 8.0 2.96-0.48mdk)] on linux2
Type "copyright", "credits" or "license" for more information.
>>> from Numeric import *
>>> from MLab import zeros
>>> joe = zeros(3,'l',1)
>>> fred = array([1,2,3],'l')
>>> joe += fred
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
TypeError: return array has incorrect type

This fails whether or not fred has the spacesaver flag set.

This, however, works fine:
>>> joe = zeros(3, 'l')
>>> fred = array([1,2,3], 'l')
>>> joe += fred
>>> joe
array([1, 2, 3])

Am I doing something wrong or is this a bug?

Please CC to me since I'm not subscribed to the list.
Thanks a lot!
-Nathan

-- 
Nathaniel Gray

California Institute of Technology
Computation and Neural Systems
--

[Numpy-discussion] Help: need info on Fortran/C mixing on Windows

[Paul F. D. <pa...@pf...>]

Any fellow nummies and Pythonistas out there with experience using Digital
Fortran on Windows?

I'm trying to get Pyfort to work with Visual Fortran / Visual C++ on
Windows. I am not usually a Windows developer so I do not know how to do
things like look at an object file to see what names are in it (as I could
do with nm on Unix). I have discovered that by including
c$DEC ATTRIBUTES C:: foo
in my Fortran, where foo is the name of my Fortran function, that my missing
external for foo disappears. This is not really ok as I want to be able to
link to unmodified Fortran files that have simply been compiled into a
foostuff.lib file.

I need information such as:

How to get the names to match up, or what name to generate in my C to
correspond to the Fortran;

Which library to load with: I've tried dfor.lib and dformd.lib, in the
context of running a python extension link through distutils, and both
complain vociferously about library conflicts.

If this was my working life I could afford to put in the time to figure all
this out, but this is pro bono publico and I would appreciate your tolerance
of my asking for help.

Even just your Makefile or setup.py from a successful project of this type
would be a big help.

Thanks,

Paul Dubois

[Numpy-discussion] RE: [Numpy-developers] Error behavior.

[Paul F. D. <pa...@pf...>]

Numpys,
This is Travis' original message to the developers list, discussing his
ideas for dealing with floating point errors. When I answered him a minute
ago I inadvertently posted to this list instead. But, it is a good idea to
have everyone's thoughts. I only ask that you keep the noise down and not
have this degenerate into the numerous IEEE discussions we have had before.

Paul

P.S. I still am looking for help on solving the Numeric bug on Alphas.

-----Original Message-----
From: num...@li...
[mailto:num...@li...]On Behalf Of Travis
Oliphant
Sent: Monday, July 23, 2001 12:46 PM
To: num...@li...
Subject: [Numpy-developers] Error behavior.



Many have noticed that with one of the newer Python releases, domain and
range errors on one of the computations in an element-by-element umath
function now always raise Python errors.

It used to be that one could say:

>>> x = arange(1,10)
>>> y = arange(0,9)
>>> print x/y

and get an output with reasonable values in positions 1..9 of the output
array but get an inf in the first position (how inf printed was platform
specific).

Many, including me, prefer returning an array with some undefined entries
rather than raising an error and ruining the rest of the perfectly valid
computations.

Proposed solutions:

1) Make a new module (bring back the old fastumathmodule), which does not
set the "check" flag when the ufunc is created so that error are not
raised --- quick and fast fix.

2) Use the warnings framework so that rather than raise a Python error, a
warning is issued.  This allows the user to do what they want with the
warning (from raise an error to ignore it).  --- this feels like a better
solution.

Please let me know your position on this, so we can take action if
warranted.

Sincerely,

Travis Oliphant



_______________________________________________
Numpy-developers mailing list
Num...@li...
http://lists.sourceforge.net/lists/listinfo/numpy-developers

[Numpy-discussion] RE: [Numpy-developers] Error behavior. (2)

[Paul F. D. <pa...@pf...>]

I understand why some people might want the behavior (return with inf or NaN
etc in arrays rather than get an exception).

Let's ask some basic questions. Having done such an operation, how do I tell
if I have any errant values in the result? For example, what can I compare
x[i] to, in order to know if it is invalid? We have had several requests for
full support of IEEE but nobody knows how to do it portably, as far as I
know.

I resist bringing back fastumath because I wonder if it really would work
portably, I don't know the answers to the simpleest questions like the
above, and if it would I know that it made people reluctant to fix bugs or
add new functions because they would have to do it twice. Adding a new
function already requires something like five different places to fix.

Using optional package MA, a user *can* achieve the goal you desire. For
example, if x and y are numpy arrays, and I want to divide them without
possibility of error, I can do:

z = MA.masked_array(x) / MA.masked_array(y)
if MA.getmask(z) is not None:
     print "Your operation failed in one or more components"

The masked_array constructor does not copy its argument. (MAs mix with
Numerics and scalars, so one of the masked_array calls above could be
omitted.)

From a Windows session:

>>> print x/y
[  1.#INF0000e+000,  2.00000000e+000,  1.50000000e+000,  1.33333333e+000,]
>>> print MA.masked_array(x)/y
[-- ,2.0 ,1.5 ,1.33333333333 ,]
>>> >>> print MA.sqrt(y-2.)
[-- ,-- ,0.0 ,1.0 ,]

I realize this isn't exactly what you had in mind, and it is more expensive
than just letting it rip, but it is reliable and portable. With the infs and
Nans, I'm not sure what you can do with the answer other than print it. It
does place the burden on the programmer to distinguish cases where he wants
to survive a divide by zero from those where he does not, but in my mind
that is a good thing.

Paul

Re: [Numpy-discussion] Troubles with arrays, proposal for Solution

[Phil A. <ph...@ge...>]

Vanroose Wim writes:
 > Dear Numerical Python Users,
 > 
 >        It is now clear to me, after the comments of Travis Oliphant, 
 > Paul Dubois and Phil Austin,  that in EVERY C-file that uses NumPy 
 > extensions, the  "import_array" statement must be present.

Actually, what I was trying to say (in a message that didn't
make it to the list) is the opposite -- import_array must be
present in only one file.

So below I would do something like: (my additions marked by PA>)

 > 
 > /////////////////////////
 > //file: arraytest.h
 > //////////////////////
 > 
 > #include "Python.h"
PA> #define PY_ARRAY_UNIQUE_SYMBOL paArray
 > #include "arrayobject.h"
 > 
 > double *myTest(void);
 > 
 > //////////////////////
 > // file: arraytest.c
 > ////////////////////
 > 
PA> #define NO_IMPORT_ARRAY
 > #include "arraytest.h"
 > 
 > double * myTest(void ){
 >   double *result;
 >   ...
 >   
 >   return result; 
 > }
 > 
 > ////////////////////
 > //file: test.c   
 > ////////////////
 > #include "arraytest.h"
 > 
 > static PyObject *function(PyObject *self, PyObject *args){
 >   
 >   ...
 >   int dimensions[2];  
 >   
 >   dimensions[0] = N;
 >   dimensions[1] = N;
 > 
 >   PyArrayObject *result ;
 >   result = (PyArrayObject *)PyArray_FromDims(2,dimensions,PyArray_DOUBLE);
 > 
 >   double *data;
 >   data = myTest();
 >   memcpy(result->data,data,N*N*sizeof(double));
 >   return PyArray_Return(result);  
 > }
 > 
 > static PyMethodDef testMethods[] = {
 >   {"test",function,1},
 >   {NULL,NULL}
 > };
 > 
 > extern "C" {
 > void  inittest(){
 >   PyObject *m,*d;
 >   m = Py_InitModule("test", testMethods);
 >   import_array();
 >   d = PyModule_GetDict(m);
 > }
 > 


You should see the following when you run /lib/cpp -E on these
files:

in arraytest.c

 #define NO_IMPORT_ARRAY
 #include "arraytest.h"

will produce a line like:

extern void **paArray;


but in  test.c

#include "arraytest.h"

will produce:

void **paArray;

and 

import_array();

will expand to


    { PyObject *numpy = PyImport_ImportModule("_numpy"); if (numpy !=
    0) { PyObject *module_dict = PyModule_GetDict(numpy); PyObject
    *c_api_object = PyDict_GetItemString(module_dict, "_ARRAY_API");
    if (((c_api_object)->ob_type == &PyCObject_Type)) { paArray =
    (void **)PyCObject_AsVoidPtr(c_api_object); } } };


The result is that there is a single global function pointer
array called paArray that has its space reserved in test.c, and
is referenced externally in arraytest.c.  This allows you to use
any numeric api call in either file.

If you compile the module to, say, test.so on unix you should see something
like this when you do a nm -D:

~phil% nm -D test.so | grep paArray
000a3d00 B paArray

i.e. one instance of paArray in the data segment of the so file.

Scanning the output of /lib/cpp -E output is a good way
to get a feel for this (on unix anyway).

Regards, Phil

[Numpy-discussion] Troubles with arrays, proposal for Solution

[Vanroose W. <van...@ru...>]

Dear Numerical Python Users,

       It is now clear to me, after the comments of Travis Oliphant, 
Paul Dubois and Phil Austin,  that in EVERY C-file that uses NumPy 
extensions, the  "import_array" statement must be present.

       However, after some try-outs with parts  my code, it seems not a 
good idea to allow a diffusion of the "import_array" statement and  
"PyArrayObject" throughout the "C" code.   The code becomes unclear by 
the mixing of Python concepts and C concepts. 

       Therefore  it is, in my opinion, a good rule to limit the use of  
NumPy to the interface between python  and "C" code; i.e. the function 
where the python parameters are read and the PyArrayObject is created.   

        Do more experienced NumPy users agree???

       The code below illustrates the limited use of NumPy parts.

Wim Vanroose

/////////////////////////
//file: arraytest.h
//////////////////////

#include "Python.h"
#include "arrayobject.h"

double *myTest(void);

//////////////////////
// file: arraytest.c
////////////////////

#include "arraytest.h"

double * myTest(void ){
  double *result;
  ...
  
  return result; 
}

////////////////////
//file: test.c   
////////////////
#include "arraytest.h"

static PyObject *function(PyObject *self, PyObject *args){
  
  ...
  int dimensions[2];  
  
  dimensions[0] = N;
  dimensions[1] = N;

  PyArrayObject *result ;
  result = (PyArrayObject *)PyArray_FromDims(2,dimensions,PyArray_DOUBLE);

  double *data;
  data = myTest();
  memcpy(result->data,data,N*N*sizeof(double));
  return PyArray_Return(result);  
}

static PyMethodDef testMethods[] = {
  {"test",function,1},
  {NULL,NULL}
};

extern "C" {
void  inittest(){
  PyObject *m,*d;
  m = Py_InitModule("test", testMethods);
  import_array();
  d = PyModule_GetDict(m);
}

Re: [Numpy-discussion] Re: Numeric.transpose (incorrect documentation)

[Tim H. <tim...@ie...>]

From: "Paul F. Dubois" <pa...@pf...>
> I will try to put a note in the transpose section if there isn't one there
> already. As noted, there are numerous reference-instead-of-a-copy returns
in
> Numeric which reflect the speed rather than safety orientation of the
> original designer. Many complaints of this nature are a complaint about
that
> choice, which made a package that was harder to understand and more
> difficult to use safely but which is about as fast as possible. I didn't
> like some of these choices but then again I didn't do the work.

Hmmm....

I tend to think that the reference semantics were the right choice, at least
for the lower level C implementation, since it's relatively easy to build
copy semantics on top of reference semantics, but not vice versa (except,
perhaps for slicing). Even now it would be pretty easy to build a pure
python ncNumeric[1], where copy was the default. One could even make
transpose do something sensible:

import ncNumeric as ncn

# C is a contiguous array
# N is a noncontiguous array

# Both of these would return a transposed copy
ncn.transpose(C)
ncn.transpose(N)

# This returns a transposed reference
ncn.transpose(C, copy=0)
# And this raises an exception.
ncn.transpose(N, copy=0)

That'd be kinda cool.

-tim

[1] nc for nonconfusing. I thought about using 'safe', but Numeric is
unlikely to ever be remotely safe....