numpy-discussion

[Numpy-discussion] zero dimensions in numarray.strings?

[Francesc A. <fa...@op...>]

Hi,

It seems like if numarray.strings module is not able to create objects when
one of its dimensions is zero. For example, this works:

>>> import numarray
>>> numarray.array([], shape=(1,0,2))
array([])

but, for strings, it raises the next error:

>>> numarray.strings.array(None, itemsize=3, shape=(1,0,2))
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 955, 
in array
    byteoffset=byteoffset, bytestride=bytestride)
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 94, 
in __init__
    self.fill(" ")
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 318, 
in fill
    self[:] = char*self._itemsize
  File "/usr/local/lib/python2.3/site-packages/numarray/generic.py", line 505, 
in _slicedIndexing
    retarr._copyFrom(value)
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 202, 
in _copyFrom
    me[:] = it[..., :self._itemsize]
  File "/usr/local/lib/python2.3/site-packages/numarray/generic.py", line 505, 
in _slicedIndexing
    retarr._copyFrom(value)
libnumarray.error: copy1bytes: access beyond buffer. offset=2 buffersize=0

I think it would be nice (and consistent) to support that in strings too.

Thanks,

-- 
Francesc Alted

Re: [Numpy-discussion] zero dimensions in numarray.strings?

[Todd M. <jm...@st...>]

On Tue, 2003-12-02 at 04:33, Francesc Alted wrote:
> Hi,
> 
> It seems like if numarray.strings module is not able to create objects when
> one of its dimensions is zero. For example, this works:
> 
> >>> import numarray
> >>> numarray.array([], shape=(1,0,2))
> array([])
> 
> but, for strings, it raises the next error:
> 
> >>> numarray.strings.array(None, itemsize=3, shape=(1,0,2))
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 955, 
> in array
>     byteoffset=byteoffset, bytestride=bytestride)
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 94, 
> in __init__
>     self.fill(" ")
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 318, 
> in fill
>     self[:] = char*self._itemsize
>   File "/usr/local/lib/python2.3/site-packages/numarray/generic.py", line 505, 
> in _slicedIndexing
>     retarr._copyFrom(value)
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 202, 
> in _copyFrom
>     me[:] = it[..., :self._itemsize]
>   File "/usr/local/lib/python2.3/site-packages/numarray/generic.py", line 505, 
> in _slicedIndexing
>     retarr._copyFrom(value)
> libnumarray.error: copy1bytes: access beyond buffer. offset=2 buffersize=0
> 
> I think it would be nice (and consistent) to support that in strings too.

I fixed the immediate problem with zero element string arrays this
morning.  I imagine there are probably other problems with that case...
Please let me know if and when you find them.


Regards,
Todd

> 
> Thanks,
-- 
Todd Miller <jm...@st...>

Re: [Numpy-discussion] zero dimensions in numarray.strings?

[Francesc A. <fa...@op...>]

A Dimarts 02 Desembre 2003 12:52, Todd Miller va escriure:
>
> I fixed the immediate problem with zero element string arrays this
> morning.  I imagine there are probably other problems with that case...
> Please let me know if and when you find them.

Thanks. Well, I've found another small issue. The next works:

>>> array([]).info()
class: <class 'numarray.numarraycore.NumArray'>
shape: (0,)
strides: (4,)
byteoffset: 0
bytestride: 4
itemsize: 4
aligned: 1
contiguous: 1
data: <memory at 40552060 with size:0 held by object 40552040 aliasing object 
00000000>
byteorder: little
byteswap: 0
type: Int32


But this don't:

>>> strings.array([])
Traceback (most recent call last):
  File "<stdin>", line 1, in ?
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 964, 
in array
    return fromlist(buffer, itemsize=itemsize, shape=shape, kind=kind)
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 828, 
in fromlist
    shape, itemsize = _slistShape(slist, itemsize=itemsize, shape=shape)
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 763, 
in _slistShape
    shape_items = _slistShape0(slist)
  File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 746, 
in _slistShape0
    maxs = _slistShape0(slist[0])
IndexError: list index out of range

Perhaps (?) you want to make it equivalent to:

>>> strings.array(None, shape=(0,), itemsize=0).info()
class: <class 'numarray.strings.CharArray'>
shape: (0,)
strides: (0,)
byteoffset: 0
bytestride: 0
itemsize: 0
aligned: 1
contiguous: 1
data: <memory at 405521e0 with size:0 held by object 405521c0 aliasing object 
00000000>
type: CharArrayType(0)

I've used the numarray present in CVS to run these test.

Cheers,

-- 
Francesc Alted

Re: [Numpy-discussion] zero dimensions in numarray.strings?

[Todd M. <jm...@st...>]

On Wed, 2003-12-03 at 05:08, Francesc Alted wrote:
> A Dimarts 02 Desembre 2003 12:52, Todd Miller va escriure:
> >
> > I fixed the immediate problem with zero element string arrays this
> > morning.  I imagine there are probably other problems with that case...
> > Please let me know if and when you find them.
> 
> Thanks. Well, I've found another small issue. The next works:
> 
> >>> array([]).info()
> class: <class 'numarray.numarraycore.NumArray'>
> shape: (0,)
> strides: (4,)
> byteoffset: 0
> bytestride: 4
> itemsize: 4
> aligned: 1
> contiguous: 1
> data: <memory at 40552060 with size:0 held by object 40552040 aliasing object 
> 00000000>
> byteorder: little
> byteswap: 0
> type: Int32
> 
> 
> But this don't:
> 
> >>> strings.array([])
> Traceback (most recent call last):
>   File "<stdin>", line 1, in ?
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 964, 
> in array
>     return fromlist(buffer, itemsize=itemsize, shape=shape, kind=kind)
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 828, 
> in fromlist
>     shape, itemsize = _slistShape(slist, itemsize=itemsize, shape=shape)
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 763, 
> in _slistShape
>     shape_items = _slistShape0(slist)
>   File "/usr/local/lib/python2.3/site-packages/numarray/strings.py", line 746, 
> in _slistShape0
>     maxs = _slistShape0(slist[0])
> IndexError: list index out of range
> 
> Perhaps (?) you want to make it equivalent to:
> 
> >>> strings.array(None, shape=(0,), itemsize=0).info()
> class: <class 'numarray.strings.CharArray'>
> shape: (0,)
> strides: (0,)
> byteoffset: 0
> bytestride: 0
> itemsize: 0
> aligned: 1
> contiguous: 1
> data: <memory at 405521e0 with size:0 held by object 405521c0 aliasing object 
> 00000000>
> type: CharArrayType(0)
> 
> I've used the numarray present in CVS to run these test.

OK.  I appended this to the 'zero dimension string' bug report so that
it doesn't get lost.

Todd

-- 
Todd Miller <jm...@st...>

[Numpy-discussion] numarray memmap

[Sebastian H. <ha...@ms...>]

Hi,
I would like to use memmap array more often - here are some comments:
(Please look for lines with $$$$)

>>> from numarray import memmap as mm
>>> m = mm.open("C:/mm1", mode='w+', len=0)
>>> n = m.insert(0,0)
>>> a = na.array(buffer=n)
X:\PrWin\wxPython\lib\PyCrust\PyCrustApp.py:1: DeprecationWarning: The
'buffer' keyword is deprecated.  Use 'sequence' instead.
Exception exceptions.AttributeError: "Memmap instance has no attribute
'_mode'" in Traceback (most recent call last):
  File "<input>", line 1, in ?
  File "X:\PrWin\numarray\numarraycore.py", line 279, in array
    type = getTypeObject(sequence, type, typecode)
  File "X:\PrWin\numarray\numarraycore.py", line 223, in getTypeObject
    return Py2NumType[ _maxtype(sequence) ]
KeyError

$$$$ Is this a mistake on my part or should the error message be "better"
???


>>> a = na.array(buffer=n, type=na.UInt16, shape=(0,0,0))
>>> a.shape
(0, 0, 0)
>>> m
<Memmap on file 'C:/mm1' with mode='w+', length=0, 1 slices>
>>> a.resize((100,100,100))
>>> a.shape
(100, 100, 100)
>>> m.sync()
>>> m.flush()
Traceback (most recent call last):
  File "<input>", line 1, in ?
  File "X:\PrWin\numarray\memmap.py", line 501, in flush
    self._consolidate(filename)
  File "X:\PrWin\numarray\memmap.py", line 469, in _consolidate
    f.write(self._buffer(m, mlen))
  File "X:\PrWin\numarray\memmap.py", line 317, in _buffer
    raise RuntimeError("Memmap no longer valid.  (closed?)")
RuntimeError: Memmap no longer valid.  (closed?)
>>> m
<Memmap on file 'C:/mm1' with mode='w+', length=0, 1 slices>

$$$$  Is this a bug ?

My idea is to "prepare" the memmap file before I actually know the array
dimensions that should go into it.  Is that a good idea ?

Thanks,
Sebastian Haase

Re: [Numpy-discussion] numarray memmap

[Todd M. <jm...@st...>]

On Wed, 2003-12-03 at 13:47, Sebastian Haase wrote:
> Hi,
> I would like to use memmap array more often - here are some comments:
> (Please look for lines with $$$$)
> 
> >>> from numarray import memmap as mm
> >>> m = mm.open("C:/mm1", mode='w+', len=0)
> >>> n = m.insert(0,0)
> >>> a = na.array(buffer=n)
> X:\PrWin\wxPython\lib\PyCrust\PyCrustApp.py:1: DeprecationWarning: The
> 'buffer' keyword is deprecated.  Use 'sequence' instead.
> Exception exceptions.AttributeError: "Memmap instance has no attribute
> '_mode'" in Traceback (most recent call last):
>   File "<input>", line 1, in ?
>   File "X:\PrWin\numarray\numarraycore.py", line 279, in array
>     type = getTypeObject(sequence, type, typecode)
>   File "X:\PrWin\numarray\numarraycore.py", line 223, in getTypeObject
>     return Py2NumType[ _maxtype(sequence) ]
> KeyError
> 
> $$$$ Is this a mistake on my part or should the error message be "better"
> ???


How's this?:

Traceback (most recent call last):
  File "sabby1.py", line 5, in ?
    a = na.array(buffer=n)
  File
"/home/jmiller/lib/python2.3/site-packages/numarray/numarraycore.py",
line 282, in array
    type = getTypeObject(sequence, type, typecode)
  File
"/home/jmiller/lib/python2.3/site-packages/numarray/numarraycore.py",
line 226, in getTypeObject
    raise TypeError("Can't determine a reasonable type from sequence")
TypeError: Can't determine a reasonable type from sequence

> 
> >>> a = na.array(buffer=n, type=na.UInt16, shape=(0,0,0))
> >>> a.shape
> (0, 0, 0)
> >>> m
> <Memmap on file 'C:/mm1' with mode='w+', length=0, 1 slices>
> >>> a.resize((100,100,100))
> >>> a.shape
> (100, 100, 100)
> >>> m.sync()
> >>> m.flush()
> Traceback (most recent call last):
>   File "<input>", line 1, in ?
>   File "X:\PrWin\numarray\memmap.py", line 501, in flush
>     self._consolidate(filename)
>   File "X:\PrWin\numarray\memmap.py", line 469, in _consolidate
>     f.write(self._buffer(m, mlen))
>   File "X:\PrWin\numarray\memmap.py", line 317, in _buffer
>     raise RuntimeError("Memmap no longer valid.  (closed?)")
> RuntimeError: Memmap no longer valid.  (closed?)
> >>> m
> <Memmap on file 'C:/mm1' with mode='w+', length=0, 1 slices>
> 
> $$$$  Is this a bug ?

Evidently.  I logged it on SF.

Switching to a length 1 map seems to "fix" the problem so I think you've
discovered an interesting edge case bug in _consolidate().  I'm still
looking for the fix.

> 
> My idea is to "prepare" the memmap file before I actually know the array
> dimensions that should go into it.  Is that a good idea ?

In theory, yes.  In reality... apparently not.

> 
> Thanks,
> Sebastian Haase

Regards,
Todd

> 
> 
> 
> 
> -------------------------------------------------------
> This SF.net email is sponsored by: SF.net Giveback Program.
> Does SourceForge.net help you be more productive?  Does it
> help you create better code?  SHARE THE LOVE, and help us help
> YOU!  Click Here: http://sourceforge.net/donate/
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
-- 
Todd Miller 			
Space Telescope Science Institute
3700 San Martin Drive
Baltimore MD, 21030
(410) 338 - 4576