Menu
▾
▴
numpy-discussion
|
From: Travis O. <oli...@ie...> - 2006-10-31 12:56:05
|
I'm recruiting more comments on python-dev regarding my two proposals for improving Python's native ability to share ndarray-like information. There is a dearth of scientific-computing and number-crunching-aware people on python-dev. The result is that I sound like a lone voice arguing for something that nobody cares about. When, I don't think that is true. Please, please. If you want Python to grow support for the array interface (or something like it), then please speak up on python-dev. Even something as simple as I really see the need for a way to exchange data-format information between two objects sharing the buffer protocol can be helpful. You can post through the gmane newsgroup interface: gmane.comp.python.devel Find any of the posts on the PEP's I've introduced. Thanks for your help. -Travis |
|
From: Alan I. <ai...@am...> - 2006-10-31 16:14:11
|
On Tue, 31 Oct 2006, Travis Oliphant wrote: > Please, please. If you want Python to grow support for > the array interface (or something like it), then please > speak up on python-dev. The easiest access to this discussion for me was http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this discussion, but I REALLY hope others will help Travis out here. (A few have.) He is fielding a lot of questions, some of which look to me to be from individuals who are ready to have fairly strong opinions without really understanding the "why" of his proposals. The good news is, there seems to be (on my naive reading) some sympathy for what Travis is trying to do. I think more motivating examples would prove helpful in swinging things. Cheers, Alan Isaac |
|
From: Travis O. <oli...@ee...> - 2006-10-31 17:04:54
|
Fernando Perez wrote: >On 10/31/06, Rich Shepard <rsh...@ap...> wrote: > > >>On Tue, 31 Oct 2006, Alan Isaac wrote: >> >> >> >>>The easiest access to this discussion for me was >>>http://news.gmane.org/group/gmane.comp.python.devel/ I cannot add to this >>>discussion, but I REALLY hope others will help Travis out here. (A few >>>have.) He is fielding a lot of questions, some of which look to me to be >>>from individuals who are ready to have fairly strong opinions without >>>really understanding the "why" of his proposals. >>> >>> >> All this is sufficiently far from my areas of expertise that I cannot >>contribute anything useful. Otherwise, I'd be happy to lend support. >> >> > >I actually worry about the same: I really would like to help, but >after reading the whole discussion, I realized that the low-level >details being asked and discussed are something I don't really know >enough to say anything. And I don't want to sound simply saying 'Hey, >Travis is great, listen to him!' to python-dev, since that (asides >from looking silly) can be somewhat counter-productive. > > >How does that sound, Travis? Is that something you think might help >you, esp. since so many of us are feeling woefully underqualified to >lend a useful hand in the actual discussion on python-dev? > > That would be great. I think a couple of things would also be useful. 1) Some way to indicate to python-dev that I'm actually speaking for more than just myself. So, while I agree that just supporting my PEP (which probably in reality needs work) without understanding it is counter-productive, a voice that says. "We really do need this kind of functionality" is at least one more voice. 2) Examples of sharing memory between two objects. PIL is the classic example and has some merit, but because the internal memory layout of the PIL is 'pointer-to-pointers' instead of 'big-chunk-of-memory' it's not a 1-1 match to NumPy and the array interface only can comunicate information about the "mode." But, I can see other examples. PyMedia, PyGame, PyVideo? CVXOPT, PyVoxel. All of these seem to define their own objects which are basically just interpretations of chunks of memory. At one time, we might have said "these should all be sub-classes of the ndarray". Now, we are thinking more along the lines of "these should all expose an array interface". The array interface is still more bulky then it needs to be (it has to go through the attribute-lookup process which can be slow). It would be much better if the extended buffer protocol were available as a function-pointer on the type object of the type. If you have an application where you've ever wanted NumPy in the core. See if the extended buffer protocol serves your purposes and if you agree, voice your approval for the PEP. In my mind, the data-format PEP does not need to go through if there really is a better way to pass data-format information through the buffer protocol. But, the extended buffer protocol we *do* need. -Travis >Regards, > >f >_______________________________________________ >SciPy-user mailing list >Sci...@sc... >http://projects.scipy.org/mailman/listinfo/scipy-user > > |
|
From: Fernando P. <fpe...@gm...> - 2006-10-31 18:18:08
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > Fernando Perez wrote: > >I actually worry about the same: I really would like to help, but > >after reading the whole discussion, I realized that the low-level > >details being asked and discussed are something I don't really know > >enough to say anything. And I don't want to sound simply saying 'Hey, > >Travis is great, listen to him!' to python-dev, since that (asides > >from looking silly) can be somewhat counter-productive. > > > > > >How does that sound, Travis? Is that something you think might help > >you, esp. since so many of us are feeling woefully underqualified to > >lend a useful hand in the actual discussion on python-dev? > > > > > > That would be great. I think a couple of things would also be useful. OK, my experience so far has been that there's a certain 'activation barrier' with Wikis, but once pages are there, people for some reason feel more comfortable filling in. So even though it's mostly a place holder, I went ahead and made this: http://www.scipy.org/ArrayInterfacePEP Hopefully as the discussion evolves, this page can be filled in with all the necessary info in one place, and it will become in a few days a solid, organized repository of all the key points in this discussion. This will be a LOT easier to refer to in the python-dev battle than randomly scattered bits of emails in the discussion thread. I'll try to spend more time on it over the next few days to fill in, but I'm pretty busy with other things as well, so hopefully others can pitch in as well. Cheers, f ps - one more thing. This guy: http://blog.vrplumber.com/ has been rewriting the OpenGL bindings using ctypes, and I've seen posts from him about numpy (in his blog). He might be able to contribute something... |
|
From: Prabhu R. <pr...@ae...> - 2006-11-01 09:53:28
|
>>>>> "Travis" == Travis Oliphant <oli...@ee...> writes:
Travis> 2) Examples of sharing memory between two objects. PIL is
Travis> the classic example and has some merit, but because the
Travis> internal memory layout of the PIL is 'pointer-to-pointers'
Travis> instead of 'big-chunk-of-memory' it's not a 1-1 match to
Travis> NumPy and the array interface only can comunicate
Travis> information about the "mode." But, I can see other
Travis> examples. PyMedia, PyGame, PyVideo? CVXOPT, PyVoxel.
VTK-Python and TVTK use this. Right now these use the buffer protocol
to allow users to pass numpy data into a vtkDataArray. It works
reasonably well. The VTK data arrays have type information so it is
not absolutely essential for an extended buffer protocol here. TVTK
uses this underlying functionality in VTK to present a cleaner
interface. The idea is to have VTK use the data setup and manipulated
by numpy without an expensive copy. The only caveat is that you need
to hang on to a reference of the numpy array if not you will end up
with a segfault.
I just thought this might be a useful data point.
FWIW, I am all for ndarrays/extended buffers in Python core. It would
encourage a standard way to pass data in and out of wrapped code.
cheers,
prabhu
|
|
From: John H. <jdh...@ac...> - 2006-10-31 17:15:18
|
>>>>> "Travis" == Travis Oliphant <oli...@ee...> writes:
Travis> All of these seem to define their own objects which are
Travis> basically just interpretations of chunks of memory. At
Travis> one time, we might have said "these should all be
Travis> sub-classes of the ndarray". Now, we are thinking more
What about blitting pixel buffers from mpl or chaco agg into various
GUI windows, GTK, Tk, WX, etc.... This seems like a ready made case
for the array interface. I could pipe in with an example like this if
it would help.
JDH
|
|
From: Charles R H. <cha...@gm...> - 2006-10-31 17:55:57
|
On 10/31/06, John Hunter <jdh...@ac...> wrote: > > >>>>> "Travis" == Travis Oliphant <oli...@ee...> writes: > Travis> All of these seem to define their own objects which are > Travis> basically just interpretations of chunks of memory. At > Travis> one time, we might have said "these should all be > Travis> sub-classes of the ndarray". Now, we are thinking more > > What about blitting pixel buffers from mpl or chaco agg into various > GUI windows, GTK, Tk, WX, etc.... This seems like a ready made case > for the array interface. I could pipe in with an example like this if > it would help. I would also like to see the socket object take a more general type than python strings. I suspect there are a lot of such things in python -- interfaces to services and such -- that would benefit from using a more general interface to memory blocks. But I am not a python developer and don't really know how to make that case. Chuck |
|
From: David L G. <Dav...@no...> - 2006-10-31 17:46:38
|
Perhaps you could please elaborate on what you regard as "the issue(s)" and "what's at stake"? I'm still comparatively new (been using python and numpy circa 8-9 months), but so far I haven't encountered any problems I'd attribute to "Python's ... [in]ability to share ndarray-like information". (The most vexing problems I've had have all been with matplotlib, its dual-mode API, and its rather clumsy and haphazard documentation.) I'd like to be supportive, but not blindly-so. DG Travis Oliphant wrote: > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > > There is a dearth of scientific-computing and number-crunching-aware > people on python-dev. The result is that I sound like a lone voice > arguing for something that nobody cares about. When, I don't think that > is true. Please, please. If you want Python to grow support for the > array interface (or something like it), then please speak up on > python-dev. > > Even something as simple as I really see the need for a way to exchange > data-format information between two objects sharing the buffer protocol > can be helpful. > > You can post through the gmane newsgroup interface: > > gmane.comp.python.devel > > Find any of the posts on the PEP's I've introduced. Thanks for your help. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > -- HMRD/ORR/NOS/NOAA <http://response.restoration.noaa.gov/emergencyresponse/> |
|
From: Sasha <nd...@ma...> - 2006-10-31 20:10:03
|
On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > I would love to help, but I feel that I will be on the other side of the disagreement. (That's why I reply here rather than on python-dev first.) I've suggested that numpy could use ctypes way to describe binary data a long time ago: http://sourceforge.net/mailarchive/message.php?msg_id=15395017 I agree that it would not work out of the box, but I don't see any major obstacles to adding the missing features. I think we should attempt to implement a ctypes approach and compare the result to the dtype object approach. If we discover any unimplementable features along the way, this will be a strong argument to abandon ctypes as a standard. The last time I looked at the issue, the only feature that ctypes were lacking for me was the itemsize information (endianness is probably lacking as well, but I did not need it.) I am not sure, however that this information belongs to the type description object rather than carried separately. Note that it is trivial to add an itemsize attribute to ctypes because each primitive ctype has a fixed itemsize and Structure can easily compute its itemsize. Doubling the number of ctypes to allow for different endianness is a more troubling proposition. In any case, I think we should seriously discuss the relative merits of the two approaches on this list and present our view to python-dev when we reach some consensus. |
|
From: Alan G I. <ai...@am...> - 2006-10-31 20:21:42
|
On Tue, 31 Oct 2006, Sasha apparently wrote:=20 > I think we should attempt to implement a ctypes approach=20 You are probably aware that Travis has addressed this at some length on comp.python.devel (for example http://article.gmane.org/gmane.comp.python.devel/84661=20 http://article.gmane.org/gmane.comp.python.devel/84670 ) but just in case ... Cheers, Alan Isaac |
|
From: Travis O. <oli...@ee...> - 2006-10-31 20:30:40
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > >>I'm recruiting more comments on python-dev regarding my two proposals >>for improving Python's native ability to share ndarray-like information. >> >> >> > >I would love to help, but I feel that I will be on the other side of >the disagreement. (That's why I reply here rather than on python-dev >first.) > > There's actually two issues here as well. 1) The extended buffer protocol 2) How to exchange data-format information through it. -Travis |
|
From: Travis O. <oli...@ee...> - 2006-10-31 20:29:46
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > >>I'm recruiting more comments on python-dev regarding my two proposals >>for improving Python's native ability to share ndarray-like information. >> >> >> > >I would love to help, but I feel that I will be on the other side of >the disagreement. (That's why I reply here rather than on python-dev >first.) > > Please read my posts about the Python type-object verses normal Python Object situation. That really is the crux of the matter. Ctypes uses a Python type object for every data-format. NumPy uses an instance of a data-type object for every data-format. What advantage do we gain by making every instance of a data-type object *also* a Python type object? We get a lot of head-ache. Have you seen what ctypes had to do? It had to define a new Dictionary object so it could attach it to the tp_dict parameter because you can't just inherit from the PyTypeObject and add the fields you want to the structure. This is my argument. I for one am not going to put any effort in that direction. People are free to do it, if they want, of course. But, it's no small change. I would, however, put effort into "undertstanding ctypes objects" as data-type objects. -Travis |
|
From: Sasha <nd...@ma...> - 2006-10-31 21:37:05
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > Please read my posts about the Python type-object verses normal Python > Object situation. That really is the crux of the matter. > I read the whole python-dev thread before replying. I may be a little biased because I never liked somewhat cryptic letter codes in Numeric and the change of codes from Numeric to numpy did not contribute to my sympathy particularly when the convert script changed all unrelated instances of 'b' in my code to something else. I am also not a big fan of record arrays. I believe that numeric data should be stored in "inverted tables," where columns of homogeneous data are stored contiguously. With this disclaimer, I will address a few issues below. > Ctypes uses a Python type object for every data-format. > NumPy uses an instance of a data-type object for every data-format. > Yes, but AFAIK this is a recent innovation. Numarray used type objects and Numeric simply used letter codes. > What advantage do we gain by making every instance of a data-type object > *also* a Python type object? I think the main advantage is that you can have instances: >>> c_int(42) c_int(42) Of course, numpy has scalars for that, but ctypes also has fixed-length arrays, that are somewhat different from ndarrays: >>> a10 = c_int*10 >>> a10() <__main__.c_int_Array_10 object at 0x2a95816958> > We get a lot of head-ache. Have you seen > what ctypes had to do? It had to define a new Dictionary object so it > could attach it to the tp_dict parameter because you can't just inherit > from the PyTypeObject and add the fields you want to the structure. > This is my argument. > But, isn't this someone else's head-ache? Someone has already gone through all these contortions, why not reuse the effort? Others on python-dev described some specific needs of ctypes that your datatype object does not address. Your point seems to be that numpy does not share these needs and could use a much simpler approach. > I for one am not going to put any effort in that direction. People are > free to do it, if they want, of course. But, it's no small change. Can you present some use cases that illustrate the advantage of your datatype approach best? I will try to implement them with ctypes. > I would, however, put effort into "undertstanding ctypes objects" as > data-type objects. Yes, this is the key. I think we should better understand ctypes limitations before proposing an alternative. At the end of the day, it is better to have buffer protocol that describes the data using ctypes types than to have no standard type information at all. |
|
From: Travis O. <oli...@ee...> - 2006-10-31 22:34:02
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > > >Yes, this is the key. I think we should better understand ctypes >limitations before proposing an alternative. > I already understand it's practical limitations --- type objects as data-type instances is too bulky and too restrictive. You have to "be a Python type object" in order to be a data-type object. I wish others would actually understand the difference. -Travis |
|
From: Travis O. <oli...@ee...> - 2006-10-31 22:48:30
|
Sasha wrote: >On 10/31/06, Travis Oliphant <oli...@ee...> wrote: > > > >>Please read my posts about the Python type-object verses normal Python >>Object situation. That really is the crux of the matter. >> >> >> >I read the whole python-dev thread before replying. I may be a little >biased because I never liked somewhat cryptic letter codes in Numeric >and the change of codes from Numeric to numpy did not contribute to my >sympathy particularly when the convert script changed all unrelated >instances of 'b' in my code to something else. > The letter codes are not data-type objects. The letter codes are only there for historical reasons (and they are in the struct and array modules too so blame Python...) >I am also not a big >fan of record arrays. I believe that numeric data should be stored in >"inverted tables," where columns of homogeneous data are stored >contiguously. > Sure. I understand this. This was my argument too when Numarray was first proposed. But, how do you memory-map a record where the data is actually stored differently? Numeric's answer is "you don't" but that is un-acceptable. This is why record-arrays were created and the whole pandora's box of data-types was opened. > > >>Ctypes uses a Python type object for every data-format. >>NumPy uses an instance of a data-type object for every data-format. >> >> >> >Yes, but AFAIK this is a recent innovation. Numarray used type >objects and Numeric simply used letter codes. > > > Recent if 1 year is recent, and only if you worry precisely about when they were Python objects. Numeric always used a PyArray_Descr * structure (which was a PyObject_HEAD away from being a Python object) to describe data. The letter codes were just simple ways to "represent" those underlying structures (which could have been exposed to Python as "data-types" from the very beginning. >>What advantage do we gain by making every instance of a data-type object >>*also* a Python type object? >> >> > >I think the main advantage is that you can have instances: > > >>>>c_int(42) >>>> >>>> >c_int(42) > >Of course, numpy has scalars for that, but ctypes also has >fixed-length arrays, that are somewhat different from ndarrays: > > Sure, but you don't "need" an instance for every memory-layout description. If you want one, then great, ctypes gives it to you. But "requiring" data-type to be encoded in a type object is over-kill. >> We get a lot of head-ache. Have you seen >>what ctypes had to do? It had to define a new Dictionary object so it >>could attach it to the tp_dict parameter because you can't just inherit >>from the PyTypeObject and add the fields you want to the structure. >>This is my argument. >> >> >> > >But, isn't this someone else's head-ache? Someone has already gone >through all these contortions, why not reuse the effort? > No, it's not that simple. We have a headache whenever we want to do something like I just did and separate out the concepts of what makes a Python Object a Python object. Now, we don't just modify a simple C-structure (PyArray_Descr *), we have to modify a "meta-type" or a altered dictionary and get that change put in to ctypes. Inheriting from Python type objects is harder. People who have apparently never tried seem to think it's not but it is. >Others on >python-dev described some specific needs of ctypes that your datatype >object does not address. Your point seems to be that numpy does not >share these needs and could use a much simpler approach. > > No, that's not the point. The data-type object could easily be extended to accomodate those needs. The point is using data-types as instances of a regular Python object or data-types as instances of a "type" object. >>I for one am not going to put any effort in that direction. People are >>free to do it, if they want, of course. But, it's no small change. >> >> > >Can you present some use cases that illustrate the advantage of your >datatype approach best? > > >I will try to implement them with ctypes. > > This is missing the point. I have no doubt ctypes "could" be used on the Python side to describe a data-type. But, we need the PyArray_Descr * structure in NumPy. What are you going to replace the PyArray_Descr * structure with? That is the point. >> I would, however, put effort into "undertstanding ctypes objects" as >>data-type objects. >> >> > >Yes, this is the key. I think we should better understand ctypes >limitations before proposing an alternative. > Please understand what I meant. I meant putting effort into getting PyArray_DescrConverter to allow ctypes inputs and convert to the appropriate PyArray_Descr * structure. I already understand ctypes objects. I want the dtype() command to also understand them. -Travis |
|
From: Alexander B. <ale...@gm...> - 2006-10-31 23:52:07
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
...
> Please understand what I meant. I meant putting effort into getting
> PyArray_DescrConverter to allow ctypes inputs and convert to the
> appropriate PyArray_Descr * structure. I already understand ctypes
> objects. I want the dtype() command to also understand them.
>
I think I am starting to understand. Forgive me for being slow.
Is it correct that you don't mind writing c_int * 10 instead of
dtype(('<i4', 10)) as long as the result does not have to be a
PyTypeObject at the C level? If this is the case, I would suggest to
merge ctypes syntax with your implementation. This may help to make
the case for the python-dev crowd. I believe very few people
understand the subtle problems in inheriting from PyTypeObject and
resist your proposal simply because they like c_int * 10 better than
dtype(('<i4', 10)).
There is no reason why ctypes should be implemented the way it is. It
is not necessary that type(c_int()) is c_int. If a type with almost
the same properties gets into the core, ctypes developers may see an
advantage of simply inheriting from it and adding a factory __call__
method. Meanwhile, users familiar with ctypes will not need to learn
yet another type specification syntax.
|
|
From: Sasha <nd...@ma...> - 2006-11-01 15:27:56
|
On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
> No, it's not that simple. We have a headache whenever we want to do
> something like I just did and separate out the concepts of what makes a
> Python Object a Python object. Now, we don't just modify a simple
> C-structure (PyArray_Descr *), we have to modify a "meta-type" or a
> altered dictionary and get that change put in to ctypes.
I think I am starting to understand. Forgive me for being slow.
Is it correct that you don't mind writing c_int * 10 instead of
dtype(('<i4', 10)) as long as the result does not have to be a
PyTypeObject at the C level? If this is the case, I would suggest to
merge ctypes syntax with your implementation. This may help to make
the case for the python-dev crowd. I believe very few people
understand the subtle problems in inheriting from PyTypeObject and
resist your proposal simply because they like c_int * 10 better than
dtype(('<i4', 10)).
There is no reason why ctypes should be implemented the way it is. It
is not necessary that type(c_int()) is c_int. If a type with almost
the same properties gets into the core, ctypes developers may see an
advantage of simply inheriting from it and adding a factory __call__
method. Meanwhile, users familiar with ctypes will not need to learn
yet another type specification syntax.
|
|
From: Alexander B. <ale...@gm...> - 2006-11-03 15:56:59
|
On 11/3/06, Torgil Svensson <tor...@gm...> wrote:
> class struct_type(Structure):
> _fields_ = [....]
>
> ...
> ... which is somewhat static in nature. How do you create "structures"
> dynamically?
>
You can put the above in a function that takes fields as an argument, or
type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b',
c_long)]})
|
|
From: Sasha <nd...@ma...> - 2006-11-04 18:03:25
|
On 11/3/06, Torgil Svensson <tor...@gm...> wrote:
> class struct_type(Structure):
> _fields_ = [....]
>
> ...
> ... which is somewhat static in nature. How do you create "structures"
> dynamically?
>
You can put the above in a function that takes fields as an argument, or
type('struct_type', (Structure,), {'_fields_':[('a', c_int), ('b',
c_long)]})
|
|
From: Bill B. <wb...@gm...> - 2006-11-01 00:26:27
|
One thing I see with PIL and Matplotlib and PyOpenGL is that they're having to add specific support for Numpy and numeric and numarray and PIL and ctypes etc. I'm also in the camp that doesn't really understand this stuff well enough to really argue in favor of it, but would the proposed extensions make it possible for all these different packages to just have to support "buffer protocol" and magically be able to work with numpy or numeric without having to introduce explicit dependencies and support code for all those other packages? If I've got the facts straight then I'd be happy to go and say that on comp.lang.python. --bb On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > > There is a dearth of scientific-computing and number-crunching-aware > people on python-dev. The result is that I sound like a lone voice > arguing for something that nobody cares about. When, I don't think that > is true. Please, please. If you want Python to grow support for the > array interface (or something like it), then please speak up on > python-dev. > > Even something as simple as I really see the need for a way to exchange > data-format information between two objects sharing the buffer protocol > can be helpful. > > You can post through the gmane newsgroup interface: > > gmane.comp.python.devel > > Find any of the posts on the PEP's I've introduced. Thanks for your help. > > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier > Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion > |
|
From: Lisandro D. <da...@gm...> - 2006-11-01 15:02:11
|
On 10/31/06, Bill Baxter <wb...@gm...> wrote: > One thing I see with PIL and Matplotlib and PyOpenGL is that they're havi= ng > to add specific support for Numpy and numeric and numarray and PIL and > ctypes etc. This is a real problem, and a waste of developing time, because we lack a **general** and **standard** mechanism. > would the proposed extensions make it > possible for all these different packages to just have to support "buffer > protocol" and magically be able to work with numpy or numeric without hav= ing > to introduce explicit dependencies and support code for all those other > packages? I think this is the actual reason of Travis's proposal, and of all other people like me supporting him. > If I've got the facts straight then I'd be happy to go and say > that on comp.lang.python. > Please, go on! --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Bill B. <wb...@gm...> - 2006-11-01 15:07:42
|
Grr, I wrote something up (it was wonderfully eloquent and convincing -- trust me!), but the gmane interface seems to have lost it (I posted two things before responding to the two "authorization requested" emails, and only one of the two things ever got posted it seems.) --bb On 11/2/06, Lisandro Dalcin <da...@gm...> wrote: > > On 10/31/06, Bill Baxter <wb...@gm...> wrote: > > One thing I see with PIL and Matplotlib and PyOpenGL is that they're > having > > to add specific support for Numpy and numeric and numarray and PIL and > > ctypes etc. > > This is a real problem, and a waste of developing time, because we > lack a **general** and **standard** mechanism. > > > would the proposed extensions make it > > possible for all these different packages to just have to support > "buffer > > protocol" and magically be able to work with numpy or numeric without > having > > to introduce explicit dependencies and support code for all those other > > packages? > > I think this is the actual reason of Travis's proposal, and of all > other people like me supporting him. > > > If I've got the facts straight then I'd be happy to go and say > > that on comp.lang.python. > > > > Please, go on! > > > |
|
From: Lisandro D. <da...@gm...> - 2006-11-01 15:48:51
|
On 10/31/06, Travis Oliphant <oli...@ie...> wrote: > > I'm recruiting more comments on python-dev regarding my two proposals > for improving Python's native ability to share ndarray-like information. > I believe there are another really important reason to support Travis's proposal. Python should have a general, simple, powerful and standard way to associate type metadata to binary streams. This enables efficient share or memory, but also can help a lot to achieve inter-language capabilities. As an example, I would to make some comments about MPI, the "de facto" standard for message-passing in parallel computing. When MPI specification was being developed, there were many different, incompatible libraries for message passing, and many of them (ie. PVM) usually communicated binary data following a pack/unpack approach. The MPI Forum decided to add an alternative approach, adding a class 'MPI_Datatype', and many predefined instances of this class (MPI_INT, MPI_FLOAT, etc.) in order to represent the basic datatypes (integers, floats, strings, complex numbers) in C, Fortran, and C++. Predefined MPI datatypes can be used as building blocks for describing very complicated memory layouts and derived,nested structures. This way, any implementations of this standard can support interprocess communication in a multi-language, muti-architecture approach. MPI datatypes can also be "decoded" in order to get the basic, predefined types contained in a user-defined datatype. I think the MPI example, despite being taken from a very specific domain with particular needs, has a strong connection with Travis's proposal. Pack/unpack degrades performance, but if you want to efficiently share binary data (with other languages/architecture), then a general and extensible mechanism for attaching type metadata to binary streams must me defined. I would like to know your opinions about this. Travis's datatypes in Python should be more that something to share data, they should be able to **describe** data. And I am not sure if ctypes can be be the way. Can Jython take advantage of ctypes??? Or in the near future CPython/Jython will have a 'jtypes' module? I forget! Travis/Pearu, why don't you develop a 'ftypes' for Fortran binary data, and next ask Python core developers to include it in standard library (a joke, of course)? Travis, if you think my previous example about MPI can help, I can resend this post to Python-Dev. I ask you this because I really want to help and not to make noise. Regars, --=20 Lisandro Dalc=EDn --------------- Centro Internacional de M=E9todos Computacionales en Ingenier=EDa (CIMEC) Instituto de Desarrollo Tecnol=F3gico para la Industria Qu=EDmica (INTEC) Consejo Nacional de Investigaciones Cient=EDficas y T=E9cnicas (CONICET) PTLC - G=FCemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 |
|
From: Travis O. <oli...@ie...> - 2006-11-01 15:55:29
|
Alexander Belopolsky wrote:
> On 10/31/06, Travis Oliphant <oli...@ee...> wrote:
> ...
>
>> Please understand what I meant. I meant putting effort into getting
>> PyArray_DescrConverter to allow ctypes inputs and convert to the
>> appropriate PyArray_Descr * structure. I already understand ctypes
>> objects. I want the dtype() command to also understand them.
>>
>>
>
> I think I am starting to understand. Forgive me for being slow.
>
And I think I'm beginning to understand why some people are resistant to
what I'm saying. I have no problem converting from ctypes types to the
underlying PyArray_Descr structure. I don't necessarily want that to be
the only way to describe new types but I sure don't mind people using
that approach.
I suspect it's the concern about "how to describe data-formats on the
Python level" that have people concerned.
> Is it correct that you don't mind writing c_int * 10 instead of
> dtype(('<i4', 10)) as long as the result does not have to be a
> PyTypeObject at the C level?
That is correct.
> If this is the case, I would suggest to
> merge ctypes syntax with your implementation.
> This may help to make
> the case for the python-dev crowd.
Thanks for this suggestion. You could very probably be right.
> I believe very few people
> understand the subtle problems in inheriting from PyTypeObject and
> resist your proposal simply because they like c_int * 10 better than
> dtype(('<i4', 10)).
>
> There is no reason why ctypes should be implemented the way it is. It
> is not necessary that type(c_int()) is c_int. If a type with almost
> the same properties gets into the core, ctypes developers may see an
> advantage of simply inheriting from it and adding a factory __call__
> method. Meanwhile, users familiar with ctypes will not need to learn
> yet another type specification syntax.
>
>
It would actually be "easier" for NumPy to understand ctypes syntax if
something like a data-type object were. As it is, the converter to a
PyArray_Descr structure will probably have to check the tp_name of the
ctypes. Maybe this will help with the PEP to describe the difficulty of
using ctypes constructs inside of other programs.
And it may be a good idea to also have a get_ctype method or some-such
on the ctypes attribute so that one could get a "ctypes description"
from the NumPy data-type.
-Travis
|
|
From: A. M. A. <per...@gm...> - 2006-11-01 18:13:17
|
On 01/11/06, Travis Oliphant <oli...@ie...> wrote: > And it may be a good idea to also have a get_ctype method or some-such > on the ctypes attribute so that one could get a "ctypes description" > from the NumPy data-type. It seems to me that at the python level, there's not much reason to choose the dtypes proposal over ctypes. There is one at the C level, it seems (though I, like perhaps most of the people on python-dev, have never actually tried using either). So perhaps, to persuade the python-dev folks, what is needed is a comparison of what has to be done at the C level. What would it take to rewrite numpy to use ctypes? There seems to be some problem with extending the type objects used by ctypes, but it's not very clear to me what that problem is (what the extensions are supposed to do). The argument that *some* datatypes format should become standard is much easier. A. M. Archibald |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X