Menu
▾
▴
numpy-discussion
|
From: Martin W. <mar...@gm...> - 2006-09-15 14:22:05
|
Hi list, I'm using PyArray_DescrConverter with a dict object to create a "struct-like" dtype from C. As the struct contains different data types I run into "unaligned access" problems. Is there a way to force alignment or to get trailing unused bytes in the dtpye? Thanks, Martin |
|
From: Martin W. <mar...@gm...> - 2006-09-15 14:23:39
|
On Friday 15 September 2006 16:13, Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a > "struct-like" dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. on IA64 I should have mentioned > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > > Thanks, Martin > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion |
|
From: Francesc A. <fa...@ca...> - 2006-09-15 14:40:48
|
A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure:
> Hi list,
>
> I'm using PyArray_DescrConverter with a dict object to create a
> "struct-like" dtype from C.
> As the struct contains different data types I run into "unaligned access"
> problems.
> Is there a way to force alignment or to get trailing unused bytes in the
> dtpye?
One possible solution is to declare void ('V' charcode) types for filling t=
he=20
gaps. For example:
In [118]: ra=3Dnumpy.rec.array("1"*300, dtype=3D[('sval','<a4'),('unused','=
|V4'),
('dval','<f8')], shape=3D2)
In [119]: ra['sval']
Out[119]:
recarray([1111, 1111],
dtype=3D'|S4')
In [120]: ra['dval']
Out[120]: array([ 9.73041595e-72, 9.73041595e-72])
You can still access the empty spaces if you want (although it is nonsense):
In [121]: ra['unused']
Out[121]:
recarray([1111, 1111],
dtype=3D'|V4')
Cheers,
=2D-=20
>0,0< Francesc Altet =A0 =A0 http://www.carabos.com/
V V C=E1rabos Coop. V. =A0=A0Enjoy Data
"-"
|
|
From: Martin W. <mar...@gm...> - 2006-09-15 15:04:54
|
On Friday 15 September 2006 16:40, Francesc Altet wrote:
> A Divendres 15 Setembre 2006 16:13, Martin Wiechert va escriure:
> > Hi list,
> >
> > I'm using PyArray_DescrConverter with a dict object to create a
> > "struct-like" dtype from C.
> > As the struct contains different data types I run into "unaligned access"
> > problems.
> > Is there a way to force alignment or to get trailing unused bytes in the
> > dtpye?
>
> One possible solution is to declare void ('V' charcode) types for filling
> the gaps. For example:
>
>
> In [118]: ra=numpy.rec.array("1"*300,
> dtype=[('sval','<a4'),('unused','|V4'), ('dval','<f8')], shape=2)
>
> In [119]: ra['sval']
> Out[119]:
> recarray([1111, 1111],
> dtype='|S4')
>
> In [120]: ra['dval']
> Out[120]: array([ 9.73041595e-72, 9.73041595e-72])
>
Hm, not exactly elegant but should work. Thanks Francesc!
Note that all but trailing gaps can be controlled using a dictionary
descriptor with an offsets key:
>>> ra = zeros ((10,), dtype = {'names': ['sval', 'dval'], 'formats': ['<a4',
'<f8'], 'offsets': [0, 8]})
>>> ra.dtype
dtype([('sval', '|S4'), ('', '|V4'), ('dval', '<f8')])
Still if anybody knows a clean solution I would be very interested.
> You can still access the empty spaces if you want (although it is
> nonsense):
>
> In [121]: ra['unused']
> Out[121]:
> recarray([1111, 1111],
> dtype='|V4')
>
> Cheers,
|
|
From: Albert S. <fu...@gm...> - 2006-09-15 15:53:14
|
Hello all
In [1]: import numpy as N
In [3]: N.dtype({'names' : ['x', 'y'],
'formats' : [N.intc, N.float64]},
align=True)
Out[3]: dtype([('x', '<i4'), ('', '|V4'), ('y', '<f8')])
The reason you might not have discovered this:
In [2]: N.dtype?
Type: type
Base Class: <type 'type'>
String Form: <type 'numpy.dtype'>
Namespace: Interactive
File: c:\python24\lib\site-packages\numpy\__init__.py
Docstring:
<no docstring>
Cheers,
Albert
> -----Original Message-----
> From: num...@li... [mailto:numpy-
> dis...@li...] On Behalf Of Martin Wiechert
> Sent: 15 September 2006 16:14
> To: numpy-discussion
> Subject: [Numpy-discussion] PyArray_DescrConverter - alignment /
> trailingunused bytes
>
> Hi list,
>
> I'm using PyArray_DescrConverter with a dict object to create a "struct-
> like"
> dtype from C.
> As the struct contains different data types I run into "unaligned access"
> problems.
> Is there a way to force alignment or to get trailing unused bytes in the
> dtpye?
>
> Thanks, Martin
|
|
From: Martin W. <mar...@gm...> - 2006-09-15 16:05:02
|
On Friday 15 September 2006 17:53, Albert Strasheim wrote:
> Hello all
>
> In [1]: import numpy as N
>
> In [3]: N.dtype({'names' : ['x', 'y'],
> 'formats' : [N.intc, N.float64]},
> align=True)
> Out[3]: dtype([('x', '<i4'), ('', '|V4'), ('y', '<f8')])
>
> The reason you might not have discovered this:
>
> In [2]: N.dtype?
> Type: type
> Base Class: <type 'type'>
> String Form: <type 'numpy.dtype'>
> Namespace: Interactive
> File: c:\python24\lib\site-packages\numpy\__init__.py
> Docstring:
> <no docstring>
>
Thanks Albert! Do you also know the corresponding C-API function? It cannot be
PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no
"align", right?
> Cheers,
>
> Albert
>
> > -----Original Message-----
> > From: num...@li... [mailto:numpy-
> > dis...@li...] On Behalf Of Martin Wiechert
> > Sent: 15 September 2006 16:14
> > To: numpy-discussion
> > Subject: [Numpy-discussion] PyArray_DescrConverter - alignment /
> > trailingunused bytes
> >
> > Hi list,
> >
> > I'm using PyArray_DescrConverter with a dict object to create a "struct-
> > like"
> > dtype from C.
> > As the struct contains different data types I run into "unaligned access"
> > problems.
> > Is there a way to force alignment or to get trailing unused bytes in the
> > dtpye?
> >
> > Thanks, Martin
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
|
|
From: Travis O. <oli...@ie...> - 2006-09-15 17:13:49
|
Martin Wiechert wrote:
> Thanks Albert! Do you also know the corresponding C-API function? It cannot be
> PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose signature has no
> "align", right?
>
The DescrConverter function is meant for "O&"-style conversions. It
can't accept an align function. We could possibly add something to the
converter to allow specification of alignment through the object to be
converted.
Or, you can just call the __new__ method of the PyArrayDescr_Type object
res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__",
"Oi", dict_object, 1))
or call the tp->new method directly:
args = Py_BuildValue("Oi", dict_object, 1);
PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL);
Py_DECREF(args);
(I think passing in NULL for the keywords is O.K., but I haven't checked
it).
-Travis
|
|
From: Martin W. <mar...@gm...> - 2006-09-15 18:01:36
|
On Friday 15 September 2006 19:14, Travis Oliphant wrote:
> Martin Wiechert wrote:
> > Thanks Albert! Do you also know the corresponding C-API function? It
> > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose
> > signature has no "align", right?
>
> The DescrConverter function is meant for "O&"-style conversions. It
> can't accept an align function. We could possibly add something to the
> converter to allow specification of alignment through the object to be
> converted.
>
I begin to see the light....
For dictionaries one could maybe just add an optional key "align".
Also an optional key "elsize" or "itemsize" to force the total size of the
record may sometimes be useful. E.g. one may want to faithfully map a given C
struct. (That's why I was asking for trailing unused bytes.)
But I understand that other things have higher priority.
> Or, you can just call the __new__ method of the PyArrayDescr_Type object
>
> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__",
> "Oi", dict_object, 1))
>
> or call the tp->new method directly:
>
> args = Py_BuildValue("Oi", dict_object, 1);
> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL);
> Py_DECREF(args);
>
Thank you! I'll try this.
> (I think passing in NULL for the keywords is O.K., but I haven't checked
> it).
>
> -Travis
>
One final question. To me the repr of a dtype with gaps is a little bit
puzzling:
>>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['<a4', '<f8', '<f4'],
'offsets': [0, 16, 24]})
dtype([('a', '|S4'), ('', '|V12'), ('b', '<f8'), ('', '|V12'), ('c', '<f4')])
There should be no gap between "b" and "c" but still the repr has ('', '|V12')
between them. Am I missing something?
Anyway, thanks again for your help.
Cheers,
Martin
>
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
|
|
From: Albert S. <fu...@gm...> - 2006-09-15 18:13:33
|
Hello all
> -----Original Message-----
> From: num...@li... [mailto:numpy-
> dis...@li...] On Behalf Of Martin Wiechert
> Sent: 15 September 2006 19:53
> To: Discussion of Numerical Python
> Subject: Re: [Numpy-discussion]PyArray_DescrConverter - alignment /
> trailingunused bytes
>
> On Friday 15 September 2006 19:14, Travis Oliphant wrote:
> > Martin Wiechert wrote:
> > > Thanks Albert! Do you also know the corresponding C-API function? It
> > > cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose
> > > signature has no "align", right?
> >
> > The DescrConverter function is meant for "O&"-style conversions. It
> > can't accept an align function. We could possibly add something to the
> > converter to allow specification of alignment through the object to be
> > converted.
> >
> <snip>
> One final question. To me the repr of a dtype with gaps is a little bit
> puzzling:
>
> >>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['<a4', '<f8', '<f4'],
> 'offsets': [0, 16, 24]})
> dtype([('a', '|S4'), ('', '|V12'), ('b', '<f8'), ('', '|V12'), ('c',
> '<f4')])
>
> There should be no gap between "b" and "c" but still the repr has ('',
> '|V12')
> between them. Am I missing something?
For performance reasons, compilers will typically align integers (and
probably floats) on 4-byte boundaries and apparently, doubles on 16-byte
boundaries.
Because compilers align like this, so does NumPy. This allows you to:
1. Take any kind of C struct definition
2. Convert it to a dtype
3. Create a NumPy array with this dtype
4. Pass the array's data pointer to C code
5. Cast the data pointer to a pointer to your C struct
6. Operate on the pointer to struct as if it were allocated in C
Cheers,
Albert
|
|
From: Albert S. <fu...@gm...> - 2006-09-15 18:17:00
|
Argh
> > <snip>
>
> > One final question. To me the repr of a dtype with gaps is a little bit
> > puzzling:
> >
> > >>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['<a4', '<f8', '<f4'],
> > 'offsets': [0, 16, 24]})
> > dtype([('a', '|S4'), ('', '|V12'), ('b', '<f8'), ('', '|V12'), ('c',
> > '<f4')])
> >
> > There should be no gap between "b" and "c" but still the repr has ('',
> > '|V12')
> > between them. Am I missing something?
I see you're not specifying align=True here. Ignore my last message.
Cheers,
Albert
|
|
From: Travis O. <oli...@ie...> - 2006-09-15 18:26:34
|
Martin Wiechert wrote:
> On Friday 15 September 2006 19:14, Travis Oliphant wrote:
>
>> Martin Wiechert wrote:
>>
>>> Thanks Albert! Do you also know the corresponding C-API function? It
>>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose
>>> signature has no "align", right?
>>>
>> The DescrConverter function is meant for "O&"-style conversions. It
>> can't accept an align function. We could possibly add something to the
>> converter to allow specification of alignment through the object to be
>> converted.
>>
>>
>
> I begin to see the light....
>
> For dictionaries one could maybe just add an optional key "align".
> Also an optional key "elsize" or "itemsize" to force the total size of the
> record may sometimes be useful. E.g. one may want to faithfully map a given C
> struct. (That's why I was asking for trailing unused bytes.)
>
> But I understand that other things have higher priority.
>
>
>> Or, you can just call the __new__ method of the PyArrayDescr_Type object
>>
>> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__",
>> "Oi", dict_object, 1))
>>
>> or call the tp->new method directly:
>>
>> args = Py_BuildValue("Oi", dict_object, 1);
>> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL);
>> Py_DECREF(args);
>>
>>
>
> Thank you! I'll try this.
>
>
>> (I think passing in NULL for the keywords is O.K., but I haven't checked
>> it).
>>
>> -Travis
>>
>>
>
> One final question. To me the repr of a dtype with gaps is a little bit
> puzzling:
>
>
>>>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['<a4', '<f8', '<f4'],
>>>>
> 'offsets': [0, 16, 24]})
> dtype([('a', '|S4'), ('', '|V12'), ('b', '<f8'), ('', '|V12'), ('c', '<f4')])
>
> There should be no gap between "b" and "c" but still the repr has ('', '|V12')
> between them. Am I missing something?
>
There was a bug I just fixed in the representation of these structures
with gaps. It should be fixed in SVN, now.
-Travis
|
|
From: Martin W. <mar...@gm...> - 2006-09-15 18:31:09
|
On Friday 15 September 2006 20:27, Travis Oliphant wrote:
> Martin Wiechert wrote:
> > On Friday 15 September 2006 19:14, Travis Oliphant wrote:
> >> Martin Wiechert wrote:
> >>> Thanks Albert! Do you also know the corresponding C-API function? It
> >>> cannot be PyArray_DescrConverter (PyObject *, PyArray_Descr **), whose
> >>> signature has no "align", right?
> >>
> >> The DescrConverter function is meant for "O&"-style conversions. It
> >> can't accept an align function. We could possibly add something to the
> >> converter to allow specification of alignment through the object to be
> >> converted.
> >
> > I begin to see the light....
> >
> > For dictionaries one could maybe just add an optional key "align".
> > Also an optional key "elsize" or "itemsize" to force the total size of
> > the record may sometimes be useful. E.g. one may want to faithfully map a
> > given C struct. (That's why I was asking for trailing unused bytes.)
> >
> > But I understand that other things have higher priority.
> >
> >> Or, you can just call the __new__ method of the PyArrayDescr_Type object
> >>
> >> res = PyObject_CallMethod((PyObject *)&PyArrayDescr_Type, "__new__",
> >> "Oi", dict_object, 1))
> >>
> >> or call the tp->new method directly:
> >>
> >> args = Py_BuildValue("Oi", dict_object, 1);
> >> PyArrayDescr_Type->tp_new(&PyArrayDescr_Type, args, NULL);
> >> Py_DECREF(args);
> >
> > Thank you! I'll try this.
> >
> >> (I think passing in NULL for the keywords is O.K., but I haven't checked
> >> it).
> >>
> >> -Travis
> >
> > One final question. To me the repr of a dtype with gaps is a little bit
> >
> > puzzling:
> >>>> dtype ({'names': ['a', 'b', 'c'], 'formats': ['<a4', '<f8', '<f4'],
> >
> > 'offsets': [0, 16, 24]})
> > dtype([('a', '|S4'), ('', '|V12'), ('b', '<f8'), ('', '|V12'), ('c',
> > '<f4')])
> >
> > There should be no gap between "b" and "c" but still the repr has ('',
> > '|V12') between them. Am I missing something?
>
> There was a bug I just fixed in the representation of these structures
> with gaps. It should be fixed in SVN, now.
>
>
> -Travis
>
>
Thanks again.
> -------------------------------------------------------------------------
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job
> easier Download IBM WebSphere Application Server v.1.0.1 based on Apache
> Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
|
|
From: Travis O. <oli...@ie...> - 2006-09-15 18:05:28
|
Martin Wiechert wrote: > Hi list, > > I'm using PyArray_DescrConverter with a dict object to create a "struct-like" > dtype from C. > As the struct contains different data types I run into "unaligned access" > problems. > Is there a way to force alignment or to get trailing unused bytes in the > dtpye? > Besides calling the tp_new function (or the method __new__), I added PyArray_DescrAlignConverter to the C-API. It is at the end and so does not require re-compilation of extension modules (unless you want to use it...) -Travis |
|
From: Martin W. <mar...@gm...> - 2006-09-15 18:12:21
|
On Friday 15 September 2006 20:05, Travis Oliphant wrote: > Martin Wiechert wrote: > > Hi list, > > > > I'm using PyArray_DescrConverter with a dict object to create a > > "struct-like" dtype from C. > > As the struct contains different data types I run into "unaligned access" > > problems. > > Is there a way to force alignment or to get trailing unused bytes in the > > dtpye? > > Besides calling the tp_new function (or the method __new__), I added > PyArray_DescrAlignConverter to the C-API. > > It is at the end and so does not require re-compilation of extension > modules (unless you want to use it...) > Thanks Travis. I appreciate your work. > -Travis > > > ------------------------------------------------------------------------- > Using Tomcat but need to do more? Need to support web services, security? > Get stuff done quickly with pre-integrated technology to make your job > easier Download IBM WebSphere Application Server v.1.0.1 based on Apache > Geronimo > http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642 > _______________________________________________ > Numpy-discussion mailing list > Num...@li... > https://lists.sourceforge.net/lists/listinfo/numpy-discussion |
×
Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.
Thanks for helping keep SourceForge clean.
X