From: Perry G. <pe...@st...> - 2002-12-28 01:22:32
|
Magnus Lie Hetland writes: > Tim Churches <tc...@op...>: > [snip] > > Just like this: > > > > >>> import Numeric > > >>> a = Numeric.array(['a','b','c'],typecode=Numeric.PyObject) > > >>> a > > array([a , b , c ],'O') > > >>> > > As you may have noticed from my previous descriptions, I'm using > numarray, not Numeric. I've used this in Numeric before -- I just > can't find the equivalent functionality in numarray :) > At the moment, PyObject arrays are not supported (mainly because it hasn't been a priority for our needs yet. But if all one needs is such an array to hold PyObjects and nothing more (for example, we envisioned more sophisticated uses such as apply object methods to the array and returning arrays of the results) than associative purposes (and being able to set and get array values), it should be quite easy to add this capability. In fact one could subclass NDArray and just define the _get and _setitem methods (I forget the exact names) and probably customize the __init__ and have the functionality that you need. I can take a look at it next week (or if you feel bold, look at NDArray yourself). As with Numeric, speed is sacrificed when using such arrays. The presumption is that one is using Numeric or numarray on such things mainly for the convenience of the array manipulations, not the kind of efficiency that bulk numerical operations provide. Combining that with RecordArrays may be a bit trickier in the sense that RecordArrays presume that records use the same buffer for all data. If one doesn't mind storing PyObject pointers in that data array, it probably is also fairly simple to extend it (but I frankly haven't thought this through so I may be wrong about how easy it is). Doing this may require some thought about how to pickle such arrays. Of course, one may have a set of arrays as Tim suggests which also acts like a record array where there is no single data buffer. Our RecordArrays were intended to map to data files closely, but other variants are certainly possible. Perry Greenfield |
From: Perry G. <pe...@st...> - 2002-12-28 19:02:22
|
> In fact, I'm thinking of adopting numarray for my pytables project, but I > don't like the fact that data is not natively aligned inside > recarrays, i.e. > there is not a gap between the different fields even if datatypes doesn't > match the "native" architecture alignement. IMO this can affect > very much to > the read/write efficency when one wants to work with data rows or > columns of > recarrays objects. > > Are there any plans to support this "natural" alignment in > addition of the non- > alignment schema present right now?. > Are you asking for an option to create record arrays with aligned fields (in the sense that the addresses of all values are consistent with their type)? Or are you arguing that non-aligned columns must be prohibited? The former is certainly possible (not not very difficult to implement; basically it requires that record sizes must be a multiple of the largest numerical type, and that padding is placed within records to ensure that all fields have offsets that are a multiple of their size). We cannot accept the latter since we need to access data that are stored in such a non-aligned manner in data files. > Francesc Alted Thanks, Perry Greenfield |
From: Francesc A. <fa...@op...> - 2002-12-29 00:59:12
|
Mensaje citado por: Perry Greenfield <pe...@st...>: > Are you asking for an option to create record arrays with > aligned fields (in the sense that the addresses of all values > are consistent with their type)? Yes, I'm advocating for that > Or are you arguing that > non-aligned columns must be prohibited? The former is certainly > possible (not not very difficult to implement; basically it requires > that record sizes must be a multiple of the largest numerical type, > and that padding is placed within records to ensure that all fields > have offsets that are a multiple of their size). Well, for the sake of keeping the size of dataset to a minimum, I think it's not necessary to adjust all the record field sizes to the largest data type because depending on the type of the field, the padding can be shorter or larger. For example, short ints only needs to be aligned in two-byte basis, while doubles need 4 bytes (or 8, I don't remember well). In any case, this depends on the architecture. But it is still possible to figure out safely what is the required minimum alignments for the different types. Look at Python's struct module for a good example on how you can reduce the padding to a minimum, without sacrificing performance. Francesc Alted |
From: Francesc A. <fa...@op...> - 2002-12-28 15:10:17
|
Mensaje citado por: Perry Greenfield <pe...@st...>: > Our RecordArrays were intended to map > to data files closely, but other variants are certainly possible. In fact, I'm thinking of adopting numarray for my pytables project, but I don't like the fact that data is not natively aligned inside recarrays, i.e. there is not a gap between the different fields even if datatypes doesn't match the "native" architecture alignement. IMO this can affect very much to the read/write efficency when one wants to work with data rows or columns of recarrays objects. Are there any plans to support this "natural" alignment in addition of the non- alignment schema present right now?. Francesc Alted |