Menu
▾
▴
numpy-discussion — Discussion list for all users of Numerical Python
You can subscribe to this list here.
| 2000 |
Jan
(8) |
Feb
(49) |
Mar
(48) |
Apr
(28) |
May
(37) |
Jun
(28) |
Jul
(16) |
Aug
(16) |
Sep
(44) |
Oct
(61) |
Nov
(31) |
Dec
(24) |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 2001 |
Jan
(56) |
Feb
(54) |
Mar
(41) |
Apr
(71) |
May
(48) |
Jun
(32) |
Jul
(53) |
Aug
(91) |
Sep
(56) |
Oct
(33) |
Nov
(81) |
Dec
(54) |
| 2002 |
Jan
(72) |
Feb
(37) |
Mar
(126) |
Apr
(62) |
May
(34) |
Jun
(124) |
Jul
(36) |
Aug
(34) |
Sep
(60) |
Oct
(37) |
Nov
(23) |
Dec
(104) |
| 2003 |
Jan
(110) |
Feb
(73) |
Mar
(42) |
Apr
(8) |
May
(76) |
Jun
(14) |
Jul
(52) |
Aug
(26) |
Sep
(108) |
Oct
(82) |
Nov
(89) |
Dec
(94) |
| 2004 |
Jan
(117) |
Feb
(86) |
Mar
(75) |
Apr
(55) |
May
(75) |
Jun
(160) |
Jul
(152) |
Aug
(86) |
Sep
(75) |
Oct
(134) |
Nov
(62) |
Dec
(60) |
| 2005 |
Jan
(187) |
Feb
(318) |
Mar
(296) |
Apr
(205) |
May
(84) |
Jun
(63) |
Jul
(122) |
Aug
(59) |
Sep
(66) |
Oct
(148) |
Nov
(120) |
Dec
(70) |
| 2006 |
Jan
(460) |
Feb
(683) |
Mar
(589) |
Apr
(559) |
May
(445) |
Jun
(712) |
Jul
(815) |
Aug
(663) |
Sep
(559) |
Oct
(930) |
Nov
(373) |
Dec
|
|
From: Keith G. <kwg...@gm...> - 2006-11-01 20:33:14
|
I had a hard time tracing a bug in my code. The culprit was this difference:
>> x
matrix([[True],
[True],
[True]], dtype=bool)
>> 1.0 - x
matrix([[ 0.],
[ 0.],
[ 0.]], dtype=float32) <------- float32
>> 1.0*x
matrix([[ 1.],
[ 1.],
[ 1.]]) <-------- float64
>> numpy.__version__
'1.0rc1'
Any chance that 1.0 - x could return dtype = float64?
|
|
From: Bill B. <wb...@gm...> - 2006-11-01 18:19:26
|
On 11/2/06, A. M. Archibald <per...@gm...> wrote: > > On 01/11/06, Travis Oliphant <oli...@ie...> wrote: > > > And it may be a good idea to also have a get_ctype method or some-such > > on the ctypes attribute so that one could get a "ctypes description" > > from the NumPy data-type. > > It seems to me that at the python level, there's not much reason to > choose the dtypes proposal over ctypes. There is one at the C level, > it seems (though I, like perhaps most of the people on python-dev, > have never actually tried using either). So perhaps, to persuade the > python-dev folks, what is needed is a comparison of what has to be > done at the C level. What would it take to rewrite numpy to use > ctypes? There seems to be some problem with extending the type objects > used by ctypes, but it's not very clear to me what that problem is > (what the extensions are supposed to do). I posted a message to the thread trying to prod things in that direction. I.e. can we see a simple concrete example of the complications involved in using ctypes interface code, vs the presumably much nicer numpy/data-descriptor code. I too think that would help convince people. Just saying it's more compilcated, trust me, doesn't help when most people reading the list have never had to write any sort of C extension. --bb |
|
From: A. M. A. <per...@gm...> - 2006-11-01 18:13:17
|
On 01/11/06, Travis Oliphant <oli...@ie...> wrote: > And it may be a good idea to also have a get_ctype method or some-such > on the ctypes attribute so that one could get a "ctypes description" > from the NumPy data-type. It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do). The argument that *some* datatypes format should become standard is much easier. A. M. Archibald |
|
From: Francesc A. <fa...@ca...> - 2006-11-01 16:29:52
|
El dc 01 de 11 del 2006 a les 15:18 +0000, en/na George Sakkis va
escriure:
> > Why this is too specific or error-prone?
>
> Because it
> 1. repeats the field types
> 2. requires adding up the length of all previous fields as offset.
>
> If you're not convinced yet, try writing this in less than 3 seconds
> ;-):
> records = N.fromfile(a_file, dtype=N.dtype('i2,i4,f4,S5,B,Q'))
> records_by_f5 = ??
Ah, I see your point :)
> > I think your solution is quite good.If what you want is a more compact way to
> > write the above, you can
> > try with:
> >
> > In [56]:records=numpy.array([(1,1),(0,2)], dtype="i2,i4")
> > In [57]:records[records['f0'].argsort()]
> > Out[57]:
> > array([(0, 2), (1, 1)],
> > dtype=[('f0', '<i2'), ('f1', '<i4')])
> > In [58]:records[records['f1'].argsort()]
> > Out[58]:
> > array([(1, 1), (0, 2)],
> > dtype=[('f0', '<i2'), ('f1', '<i4')])
>
> Ah, much better; I didn't know you can index a normal array (not
> recarray) by label. Now, if there's a way to do the sorting in place
> (records.sort('f1') doesn't work unfortunately), that would be perfect.
Yes, I agree that having the possibility to do records.sort('f1') would
be a great addition (both in terms of usability but also efficiency).
Cheers,
--
Francesc Altet | Be careful about using the following code --
Carabos Coop. V. | I've only proven that it works,
www.carabos.com | I haven't tested it. -- Donald Knuth
|
|
From: Travis O. <oli...@ie...> - 2006-11-01 15:55:29
|
Alexander Belopolsky wrote:
> On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
> ...
>
>> Please understand what I meant. I meant putting effort into getting
>> PyArray_DescrConverter to allow ctypes inputs and convert to the
>> appropriate PyArray_Descr * structure. I already understand ctypes
>> objects. I want the dtype() command to also understand them.
>>
>>
>
> I think I am starting to understand. Forgive me for being slow.
>
And I think I'm beginning to understand why some people are resistant to
what I'm saying. I have no problem converting from ctypes types to the
underlying PyArray_Descr structure. I don't necessarily want that to be
the only way to describe new types but I sure don't mind people using
that approach.
I suspect it's the concern about "how to describe data-formats on the
Python level" that have people concerned.
> Is it correct that you don't mind writing c_int * 10 instead of
> dtype(('<i4', 10)) as long as the result does not have to be a
> PyTypeObject at the C level?
That is correct.
> If this is the case, I would suggest to
> merge ctypes syntax with your implementation.
> This may help to make
> the case for the python-dev crowd.
Thanks for this suggestion. You could very probably be right.
> I believe very few people
> understand the subtle problems in inheriting from PyTypeObject and
> resist your proposal simply because they like c_int * 10 better than
> dtype(('<i4', 10)).
>
> There is no reason why ctypes should be implemented the way it is. It
> is not necessary that type(c_int()) is c_int. If a type with almost
> the same properties gets into the core, ctypes developers may see an
> advantage of simply inheriting from it and adding a factory __call__
> method. Meanwhile, users familiar with ctypes will not need to learn
> yet another type specification syntax.
>
>
It would actually be "easier" for NumPy to understand ctypes syntax if
something like a data-type object were. As it is, the converter to a
PyArray_Descr structure will probably have to check the tp_name of the
ctypes. Maybe this will help with the PEP to describe the difficulty of
using ctypes constructs inside of other programs.
And it may be a good idea to also have a get_ctype method or some-such
on the ctypes attribute so that one could get a "ctypes description"
from the NumPy data-type.
-Travis
|
|
From: Lisandro D. <da...@gm...> - 2006-11-01 15:48:51
|
On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > I believe there are another really important reason to support Travis's proposal. Python should have a general, simple, powerful and standard way to associate type metadata to binary streams. This enables efficient share or memory, but also can help a lot to achieve inter-language capabilities. As an example, I would to make some comments about MPI, the "de facto" standard for message-passing in parallel computing. When MPI specification was being developed, there were many different, incompatible libraries for message passing, and many of them (ie. PVM) usually communicated binary data following a pack/unpack approach. The MPI Forum decided to add an alternative approach, adding a class 'MPI_Datatype', and many predefined instances of this class (MPI_INT, MPI_FLOAT, etc.) in order to represent the basic datatypes (integers, floats, strings, complex numbers) in C, Fortran, and C++. Predefined MPI datatypes can be used as building blocks for describing very complicated memory layouts and derived,nested structures. This way, any implementations of this standard can support interprocess communication in a multi-language, muti-architecture approach. MPI datatypes can also be "decoded" in order to get the basic, predefined types contained in a user-defined datatype. I think the MPI example, despite being taken from a very specific domain with particular needs, has a strong connection with Travis's proposal. Pack/unpack degrades performance, but if you want to efficiently share binary data (with other languages/architecture), then a general and extensible mechanism for attaching type metadata to binary streams must me defined. I would like to know your opinions about this. Travis's datatypes in Python should be more that something to share data, they should be able to **describe** data. And I am not sure if ctypes can be be the way. Can Jython take advantage of ctypes??? Or in the near future CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu, why don't you develop a 'ftypes' for Fortran binary data, and next ask Python core developers to include it in standard library (a joke, of course)? Travis, if you think my previous example about MPI can help, I can resend this post to Python-Dev. I ask you this because I really want to help and not to make noise. Regars, --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Sasha <nd...@ma...> - 2006-11-01 15:27:56
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
> No, it's not that simple. We have a headache whenever we want to do
> something like I just did and separate out the concepts of what makes a
> Python Object a Python object. Now, we don't just modify a simple
> C-structure (PyArray_Descr *), we have to modify a "meta-type" or a
> altered dictionary and get that change put in to ctypes.
I think I am starting to understand. Forgive me for being slow.
Is it correct that you don't mind writing c_int * 10 instead of
dtype(('<i4', 10)) as long as the result does not have to be a
PyTypeObject at the C level? If this is the case, I would suggest to
merge ctypes syntax with your implementation. This may help to make
the case for the python-dev crowd. I believe very few people
understand the subtle problems in inheriting from PyTypeObject and
resist your proposal simply because they like c_int * 10 better than
dtype(('<i4', 10)).
There is no reason why ctypes should be implemented the way it is. It
is not necessary that type(c_int()) is c_int. If a type with almost
the same properties gets into the core, ctypes developers may see an
advantage of simply inheriting from it and adding a factory __call__
method. Meanwhile, users familiar with ctypes will not need to learn
yet another type specification syntax.
|
|
From: George S. <geo...@gm...> - 2006-11-01 15:18:51
|
Francesc Altet wrote:
> El dt 31 de 10 del 2006 a les 23:38 +0000, en/na George Sakkis va
> escriure:
> > Is there a more elegant and/or faster way to read some records from a
> > file and then sort them by different fields ? What I have now is too
> > specific and error-prone in general:
> >
> > import numpy as N
> > records = N.fromfile(a_file, dtype=N.dtype('i2,i4'))
> > records_by_f0 = records.take(records.getfield('i2').argsort())
> > records_by_f1 = records.take(records.getfield('i4',2).argsort())
> >
> > If there's a better way, I'd like to see it; bonus points for in-place
> > sorting.
>
> Why this is too specific or error-prone?
Because it
1. repeats the field types
2. requires adding up the length of all previous fields as offset.
If you're not convinced yet, try writing this in less than 3 seconds
;-):
records = N.fromfile(a_file, dtype=N.dtype('i2,i4,f4,S5,B,Q'))
records_by_f5 = ??
> I think your solution is quite good.If what you want is a more compact way to
> write the above, you can
> try with:
>
> In [56]:records=numpy.array([(1,1),(0,2)], dtype="i2,i4")
> In [57]:records[records['f0'].argsort()]
> Out[57]:
> array([(0, 2), (1, 1)],
> dtype=[('f0', '<i2'), ('f1', '<i4')])
> In [58]:records[records['f1'].argsort()]
> Out[58]:
> array([(1, 1), (0, 2)],
> dtype=[('f0', '<i2'), ('f1', '<i4')])
Ah, much better; I didn't know you can index a normal array (not
recarray) by label. Now, if there's a way to do the sorting in place
(records.sort('f1') doesn't work unfortunately), that would be perfect.
Thanks,
George
|
|
From: Bill B. <wb...@gm...> - 2006-11-01 15:07:42
|
Grr, I wrote something up (it was wonderfully eloquent and convincing -- trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.) --bb On 11/2/06, Lisandro Dalcin <da...@gm...> wrote: > > On 10/31/06, Bill Baxter <wb...@gm...> wrote: > > One thing I see with PIL and Matplotlib and PyOpenGL is that they're > having > > to add specific support for Numpy and numeric and numarray and PIL and > > ctypes etc. > > This is a real problem, and a waste of developing time, because we > lack a **general** and **standard** mechanism. > > > would the proposed extensions make it > > possible for all these different packages to just have to support > "buffer > > protocol" and magically be able to work with numpy or numeric without > having > > to introduce explicit dependencies and support code for all those other > > packages? > > I think this is the actual reason of Travis's proposal, and of all > other people like me supporting him. > > > If I've got the facts straight then I'd be happy to go and say > > that on comp.lang.python. > > > > Please, go on! > > > |
|
From: Lisandro D. <da...@gm...> - 2006-11-01 15:02:11
|
On 10/31/06, Bill Baxter <wb...@gm...> wrote: > One thing I see with PIL and Matplotlib and PyOpenGL is that they're havi= ng > to add specific support for Numpy and numeric and numarray and PIL and > ctypes etc. This is a real problem, and a waste of developing time, because we lack a **general** and **standard** mechanism. > would the proposed extensions make it > possible for all these different packages to just have to support "buffer > protocol" and magically be able to work with numpy or numeric without hav= ing > to introduce explicit dependencies and support code for all those other > packages? I think this is the actual reason of Travis's proposal, and of all other people like me supporting him. > If I've got the facts straight then I'd be happy to go and say > that on comp.lang.python. > Please, go on! --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Francesc A. <fa...@ca...> - 2006-11-01 12:03:19
|
El dt 31 de 10 del 2006 a les 23:38 +0000, en/na George Sakkis va
escriure:
> Is there a more elegant and/or faster way to read some records from a
> file and then sort them by different fields ? What I have now is too
> specific and error-prone in general:
>
> import numpy as N
> records = N.fromfile(a_file, dtype=N.dtype('i2,i4'))
> records_by_f0 = records.take(records.getfield('i2').argsort())
> records_by_f1 = records.take(records.getfield('i4',2).argsort())
>
> If there's a better way, I'd like to see it; bonus points for in-place
> sorting.
Why this is too specific or error-prone? I think your solution is quite
good. If what you want is a more compact way to write the above, you can
try with:
In [56]:records=numpy.array([(1,1),(0,2)], dtype="i2,i4")
In [57]:records[records['f0'].argsort()]
Out[57]:
array([(0, 2), (1, 1)],
dtype=[('f0', '<i2'), ('f1', '<i4')])
In [58]:records[records['f1'].argsort()]
Out[58]:
array([(1, 1), (0, 2)],
dtype=[('f0', '<i2'), ('f1', '<i4')])
HTH,
--
Francesc Altet | Be careful about using the following code --
Carabos Coop. V. | I've only proven that it works,
www.carabos.com | I haven't tested it. -- Donald Knuth
|
|
From: Prabhu R. <pr...@ae...> - 2006-11-01 09:53:28
|
>>>>> "Travis" == Travis Oliphant <oli...@ee...> writes:
Travis> 2) Examples of sharing memory between two objects. PIL is
Travis> the classic example and has some merit, but because the
Travis> internal memory layout of the PIL is 'pointer-to-pointers'
Travis> instead of 'big-chunk-of-memory' it's not a 1-1 match to
Travis> NumPy and the array interface only can comunicate
Travis> information about the "mode." But, I can see other
Travis> examples. PyMedia, PyGame, PyVideo? CVXOPT, PyVoxel.
VTK-Python and TVTK use this. Right now these use the buffer protocol
to allow users to pass numpy data into a vtkDataArray. It works
reasonably well. The VTK data arrays have type information so it is
not absolutely essential for an extended buffer protocol here. TVTK
uses this underlying functionality in VTK to present a cleaner
interface. The idea is to have VTK use the data setup and manipulated
by numpy without an expensive copy. The only caveat is that you need
to hang on to a reference of the numpy array if not you will end up
with a segfault.
I just thought this might be a useful data point.
FWIW, I am all for ndarrays/extended buffers in Python core. It would
encourage a standard way to pass data in and out of wrapped code.
cheers,
prabhu
|
|
From: Erin S. <eri...@gm...> - 2006-11-01 04:02:53
|
On 10/31/06, Brian Granger <ell...@gm...> wrote: > Nice binaries for gfortran can be found here: > > hpc.sourceforge.net/ Do you know how to make macports recognize this compiler if it is outside the macports tree? I had a version of gfortran (4.2.0) already on my system, but it insisted on compiling its own version (4.1.1) Erin |
|
From: Brian G. <ell...@gm...> - 2006-11-01 00:46:39
|
Nice binaries for gfortran can be found here: hpc.sourceforge.net/ On 10/31/06, Erin Sheldon <eri...@gm...> wrote: > Hi Alan - > > I have not had luck with the binary distros. There is always > something that doesn't work, so I will be interested in > the results of your efforts. > > The biggest problem with compiling things yourself is going > to be dealing with LAPACK and fortran issues, and the > backends in matplotlib. > > FWIW, the most stable solution I have found that doesn't involve > dealing with dependencies yourself is darwinports (now > macports). It currently has 1.0b5 numpy and scipy 0.5.1, which > depend on python 2.4. Matplotlib worked with the wxPython > backend but not others. While not the latest versions it will get > you started, and I would guess 1.0 will be available soon (anyone?). > I had absolutely zero problems using this once I realized that I > had to use wxPython for the plotting backend. The snag > is that gfortran is needed, which requires compiling gcc 4.0 which > took 6 hours on my powerbook, so you will need some patience. > > Erin > > On 10/31/06, Steve Lianoglou <lis...@ar...> wrote: > > Hi Alan, > > > > > > > I installed Python 2.5 > > > from python.org painlessly, but it looks like > > > I'll have to compile numpy from source. > > > Do I understand that right? > > > > I believe you'll have to if you want to use Python 2.5. > > > > > ... so I am hoping someone patient > > > will point me to or lead me through the steps I need > > > to take to successfully install numpy on this platform. > > > > I think the instructions on the scipy wiki are pretty thorough: > > http://scipy.org/Installing_SciPy/Mac_OS_X > > > > One thing to note is that I think I've seen on this list that people > > have been having problems compiling and using num/scipy on Python 2.5 > > (on any platform (?)) ... I haven't tried it myself, but that's the > > general feeling I've gotten from some of the emails on that subject. > > I could be wrong though. > > > > Another thing to consider would be to install the "Superpack" which > > is linked to in the mac section from the scipy downloads page: > > http://www.scipy.org/Download > > > > I also remember seeing on the ML that there were some problems w/ > > that superpack it not including some config file for ipython or > > something, but I'm not sure. > > > > For what it's worth, I'm using MacPorts (http://macports.org) to > > manage my python (and a other) installations, and have been compiling > > scipy/numpy/matplotlib/ipython from svn checkouts every now and > > again ... this has been working well for me. The version of python > > you get from macports is 2.4.3 > > > > Hope that helps. > > -steve > > > > > > ------------------------------------------------------------------------- > > Using Tomcat but need to do more? Need to support web services, security? > > Get stuff done quickly with pre-integrated technology to make your job easier > > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > > _______________________________________________ > > Numpy-discussion mailing list > > Num...@li... > > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > |
|
From: Bill B. <wb...@gm...> - 2006-11-01 00:26:27
|
One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc. I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages? If I've got the facts straight then I'd be happy to go and say that on comp.lang.python. --bb On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > > There is a dearth of scientific-computing and number-crunching-aware > people on python-dev. The result is that I sound like a lone voice > arguing for something that nobody cares about. When, I don't think that > is true. Please, please. If you want Python to grow support for the > array interface (or something like it), then please speak up on > python-dev. > > Even something as simple as I really see the need for a way to exchange > data-format information between two objects sharing the buffer protocol > can be helpful. > > You can post through the gmane newsgroup interface: > > gmane.comp.python.devel > > Find any of the posts on the PEP's I've introduced. Thanks for your help. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > |
|
From: Alexander B. <ale...@gm...> - 2006-10-31 23:52:07
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
...
> Please understand what I meant. I meant putting effort into getting
> PyArray_DescrConverter to allow ctypes inputs and convert to the
> appropriate PyArray_Descr * structure. I already understand ctypes
> objects. I want the dtype() command to also understand them.
>
I think I am starting to understand. Forgive me for being slow.
Is it correct that you don't mind writing c_int * 10 instead of
dtype(('<i4', 10)) as long as the result does not have to be a
PyTypeObject at the C level? If this is the case, I would suggest to
merge ctypes syntax with your implementation. This may help to make
the case for the python-dev crowd. I believe very few people
understand the subtle problems in inheriting from PyTypeObject and
resist your proposal simply because they like c_int * 10 better than
dtype(('<i4', 10)).
There is no reason why ctypes should be implemented the way it is. It
is not necessary that type(c_int()) is c_int. If a type with almost
the same properties gets into the core, ctypes developers may see an
advantage of simply inheriting from it and adding a factory __call__
method. Meanwhile, users familiar with ctypes will not need to learn
yet another type specification syntax.
|
|
From: George S. <geo...@gm...> - 2006-10-31 23:38:46
|
Is there a more elegant and/or faster way to read some records from a
file and then sort them by different fields ? What I have now is too
specific and error-prone in general:
import numpy as N
records = N.fromfile(a_file, dtype=N.dtype('i2,i4'))
records_by_f0 = records.take(records.getfield('i2').argsort())
records_by_f1 = records.take(records.getfield('i4',2).argsort())
If there's a better way, I'd like to see it; bonus points for in-place
sorting.
Thanks,
George
|
|
From: Josh M. <jos...@gm...> - 2006-10-31 22:51:29
|
On 11/31/06, Fernando Perez <fpe...@gm...> wrote: > Fernando Perez wrote: > ps - one more thing. This guy: > > http://blog.vrplumber.com/ > > has been rewriting the OpenGL bindings using ctypes, and I've seen > posts from him about numpy (in his blog). He might be able to > contribute something... I've been working on the OSX port and extensions for OpenGL-ctypes. (now released as PyOpenGL-3.00a, please test!). What Mike has done is define setuptools plugins to interface to different kinds of array data. The default type is ctypes arrays, or numpy arrays if numpy is installed. The data types handled are ctypes sized arrays, ctypes pointers, strings (read-only), and Python lists. (and old Numeric/ numarray via a non-default build.) Summaries of development and usage, resp, are found: http://pyopengl.sourceforge.net/ctypes/development.html http://pyopengl.sourceforge.net/ctypes/using.html Notes on array handling from the above: > Perhaps the most complex mechanisms in OpenGL-ctypes are those > which implement the array-based operations which allow for using > low-level blocks of formatted data to communicate with the OpenGL > implementation. OpenGL-ctypes preferred basic array implementation > is the (new) numpy reimplementation of the original Numeric Python. > > The array handling functionality provided within OpenGL-ctypes is > localised to the OpenGL.arrays sub-package. Within the package, > there are two major classes, one (the FormatHandler) which > implements an interface to a way of storing data in Python, and > another (the ArrayDatatype) which models an OpenGL array format. > The ArrayDatatype classes use FormatHandlers to manipulate array- > compatible objects for use in the system. and on FormatHandlers, for each of the datatypes I mentioned earlier: > Each format handler is responsible for implementing an API that > ArrayDatatypes can use to work with the Python data-format. Data- > formats can support a subset of the API, they only need to support > those aspects of the data-format which make sense. Now, I haven't spent much time looking at these parts of OpenGL- ctypes, as they have just worked for me. I would think that it would be trivial to write a FormatHandler which uses the ndarray interface and data type description to use any object implenting it as an input for OpenGL. This would include things such as PIL images. Mike, can you give us your opinion on how a standardised data type descriptor would be helpful for PyOpenGL? The PEP and some information about it can be found here: http://www.scipy.org/ArrayInterfacePEP Cheers, Josh |
|
From: Travis O. <oli...@ee...> - 2006-10-31 22:48:30
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > > > >>Please read my posts about the Python type-object verses normal Python >>Object situation. That really is the crux of the matter. >> >> >> >I read the whole python-dev thread before replying. I may be a little >biased because I never liked somewhat cryptic letter codes in Numeric >and the change of codes from Numeric to numpy did not contribute to my >sympathy particularly when the convert script changed all unrelated >instances of 'b' in my code to something else. > The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...) >I am also not a big >fan of record arrays. I believe that numeric data should be stored in >"inverted tables," where columns of homogeneous data are stored >contiguously. > Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is "you don't" but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened. > > >>Ctypes uses a Python type object for every data-format. >>NumPy uses an instance of a data-type object for every data-format. >> >> >> >Yes, but AFAIK this is a recent innovation. Numarray used type >objects and Numeric simply used letter codes. > > > Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to "represent" those underlying structures (which could have been exposed to Python as "data-types" from the very beginning. >>What advantage do we gain by making every instance of a data-type object >>*also* a Python type object? >> >> > >I think the main advantage is that you can have instances: > > >>>>c_int(42) >>>> >>>> >c_int(42) > >Of course, numpy has scalars for that, but ctypes also has >fixed-length arrays, that are somewhat different from ndarrays: > > Sure, but you don't "need" an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But "requiring" data-type to be encoded in a type object is over-kill. >> We get a lot of head-ache. Have you seen >>what ctypes had to do? It had to define a new Dictionary object so it >>could attach it to the tp_dict parameter because you can't just inherit >>from the PyTypeObject and add the fields you want to the structure. >>This is my argument. >> >> >> > >But, isn't this someone else's head-ache? Someone has already gone >through all these contortions, why not reuse the effort? > No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a "meta-type" or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is. >Others on >python-dev described some specific needs of ctypes that your datatype >object does not address. Your point seems to be that numpy does not >share these needs and could use a much simpler approach. > > No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a "type" object. >>I for one am not going to put any effort in that direction. People are >>free to do it, if they want, of course. But, it's no small change. >> >> > >Can you present some use cases that illustrate the advantage of your >datatype approach best? > > >I will try to implement them with ctypes. > > This is missing the point. I have no doubt ctypes "could" be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point. >> I would, however, put effort into "undertstanding ctypes objects" as >>data-type objects. >> >> > >Yes, this is the key. I think we should better understand ctypes >limitations before proposing an alternative. > Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis |
|
From: Travis O. <oli...@ee...> - 2006-10-31 22:34:02
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > > >Yes, this is the key. I think we should better understand ctypes >limitations before proposing an alternative. > I already understand it's practical limitations --- type objects as data-type instances is too bulky and too restrictive. You have to "be a Python type object" in order to be a data-type object. I wish others would actually understand the difference. -Travis |
|
From: Sasha <nd...@ma...> - 2006-10-31 21:37:05
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > Please read my posts about the Python type-object verses normal Python > Object situation. That really is the crux of the matter. > I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously. With this disclaimer, I will address a few issues below. > Ctypes uses a Python type object for every data-format. > NumPy uses an instance of a data-type object for every data-format. > Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes. > What advantage do we gain by making every instance of a data-type object > *also* a Python type object? I think the main advantage is that you can have instances: >>> c_int(42) c_int(42) Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays: >>> a10 = c_int*10 >>> a10() <__main__.c_int_Array_10 object at 0x2a95816958> > We get a lot of head-ache. Have you seen > what ctypes had to do? It had to define a new Dictionary object so it > could attach it to the tp_dict parameter because you can't just inherit > from the PyTypeObject and add the fields you want to the structure. > This is my argument. > But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach. > I for one am not going to put any effort in that direction. People are > free to do it, if they want, of course. But, it's no small change. Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes. > I would, however, put effort into "undertstanding ctypes objects" as > data-type objects. Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. At the end of the day, it is better to have buffer protocol that describes the data using ctypes types than to have no standard type information at all. |
|
From: Crystal X. <Bea...@gm...> - 2006-10-31 21:30:46
|
Odis Singleton |
|
From: Travis O. <oli...@ee...> - 2006-10-31 20:30:40
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > >>I'm recruiting more comments on python-dev regarding my two proposals >>for improving Python's native ability to share ndarray-like information. >> >> >> > >I would love to help, but I feel that I will be on the other side of >the disagreement. (That's why I reply here rather than on python-dev >first.) > > There's actually two issues here as well. 1) The extended buffer protocol 2) How to exchange data-format information through it. -Travis |
|
From: Travis O. <oli...@ee...> - 2006-10-31 20:29:46
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > >>I'm recruiting more comments on python-dev regarding my two proposals >>for improving Python's native ability to share ndarray-like information. >> >> >> > >I would love to help, but I feel that I will be on the other side of >the disagreement. (That's why I reply here rather than on python-dev >first.) > > Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into "undertstanding ctypes objects" as data-type objects. -Travis |
|
From: Alan G I. <ai...@am...> - 2006-10-31 20:21:42
|
On Tue, 31 Oct 2006, Sasha apparently wrote:=20 > I think we should attempt to implement a ctypes approach=20 You are probably aware that Travis has addressed this at some length on comp.python.devel (for example http://article.gmane.org/gmane.comp.python.devel/84661=20 http://article.gmane.org/gmane.comp.python.devel/84670 ) but just in case ... Cheers, Alan Isaac |
258 messages has been excluded from this view by a project administrator.
