numpy-discussion

[Numpy-discussion] None as missing value

[Keith G. <kwg...@gm...>]

I have a list x

>> x
[[1, None], [2, 3]]

that I generate outside of numpy (with plain python). What is the best
way to convert x into an array? This doesn't work

>> asarray(x)

array([[1, None],
       [2, 3]], dtype=object)   <-- I'm hoping for something like dtype=float64

Is there something better than None to represent missing values so
that when I convert to numpy arrays (actually matrices) I'll be all
set? (I could use -99, but that would be even more embarrassing than
my python skills.)

If there is nothing better than None, what's a fast way to take care
of the None's if x is faily large?

Re: [Numpy-discussion] None as missing value

[Tim L. <tim...@gm...>]

On 7/3/06, Keith Goodman <kwg...@gm...> wrote:
> I have a list x
>
> >> x
> [[1, None], [2, 3]]
>
> that I generate outside of numpy (with plain python). What is the best
> way to convert x into an array? This doesn't work
>
> >> asarray(x)
>
> array([[1, None],
>        [2, 3]], dtype=object)   <-- I'm hoping for something like dtype=float64
>
> Is there something better than None to represent missing values so
> that when I convert to numpy arrays (actually matrices) I'll be all
> set? (I could use -99, but that would be even more embarrassing than
> my python skills.)
>
> If there is nothing better than None, what's a fast way to take care
> of the None's if x is faily large?

You might want to have a look at the masked array module in numpy
(numpy.ma). The following example might help to get you started.

>>> import numpy as N
>>> x = [[1, None], [2, 3]]
>>> m = N.ma.array(x, mask=N.equal(x, None))
>>> print m
[[1 --]
 [2 3]]


Cheers,

Tim

>
> Using Tomcat but need to do more? Need to support web services, security?
> Get stuff done quickly with pre-integrated technology to make your job easier
> Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
> _______________________________________________
> Numpy-discussion mailing list
> Num...@li...
> https://lists.sourceforge.net/lists/listinfo/numpy-discussion
>

Re: [Numpy-discussion] None as missing value

[Pierre GM <pgm...@ma...>]

Keith,

> > Is there something better than None to represent missing values so
> > that when I convert to numpy arrays (actually matrices) I'll be all
> > set? (I could use -99, but that would be even more embarrassing than
> > my python skills.)

As Tim suggested, have a look at the masked array module. However, the result 
will NOT be exportable to matrices, unless you fill the missing value first 
(for example, with -99 ;)). I use MaskedArrays a lot, they're quite flexible.

An alternative would be to use nan instead of None:
>>> import numpy as N
>>> x = [[1, nan], [2, 3]]
>>> print N.matrix(x)
[[ 1.                 nan]
 [ 2.          3.        ]]

Of course, the solution will depend on what you need...

Re: [Numpy-discussion] None as missing value

[Travis O. <oli...@ie...>]

Keith Goodman wrote:
> I have a list x
>
>   
>>> x
>>>       
> [[1, None], [2, 3]]
>
> that I generate outside of numpy (with plain python). What is the best
> way to convert x into an array? This doesn't work
>
>   
>>> asarray(x)
>>>       
>
> array([[1, None],
>        [2, 3]], dtype=object)   <-- I'm hoping for something like dtype=float64
>
> Is there something better than None to represent missing values so
> that when I convert to numpy arrays (actually matrices) I'll be all
> set? (I could use -99, but that would be even more embarrassing than
> my python skills.)
>   

You can use a masked array specifically, or use nan's for missing values 
and just tell Python you want a floating-point array (because it finds 
the None object it's guessing incorrectly you want an "object" array.

asarray(x, dtype=float)

array([[ 1.        ,         nan],
       [ 2.        ,  3.        ]])


-Travis

Re: [Numpy-discussion] None as missing value

[Pierre GM <pgm...@ma...>]

> I was also a bit surprised at the following behavior:
>>> a = numpy.asarray([1,1])
>>> a
array([1, 1])
>>> a[0]=numpy.nan
>>> a
array([0, 1])

Seems to affect only the int_ arrays:

>>> a = numpy.asarray([1,1], dtype=float_)
>>> a
array([1., 1.])
>>> a[0]=numpy.nan
>>> a
array([        nan,  1.        ])

Re: [Numpy-discussion] None as missing value

[Sven S. <sve...@gm...>]

Pierre GM schrieb:
>> I was also a bit surprised at the following behavior:
>>>> a = numpy.asarray([1,1])
>>>> a
> array([1, 1])
>>>> a[0]=numpy.nan
>>>> a
> array([0, 1])
> 
> Seems to affect only the int_ arrays:
> 
>>>> a = numpy.asarray([1,1], dtype=float_)
>>>> a
> array([1., 1.])
>>>> a[0]=numpy.nan
>>>> a
> array([        nan,  1.        ])
> 

Sure it works with floats. The question is, should there maybe be an
error if the type of the assigned value doesn't match the dtype of the
array, instead of silently doing something unexpected?
Consider the following (with still the same *integer* array a from my
post above):

>>> a[0]=0.1
>>> a
array([0, 1])
>>> a[0]='0'
Traceback (most recent call last):
  File "<interactive input>", line 1, in ?
TypeError: an integer is required

This TypeError makes a lot of sense, but why does it only react to
strings, not to floats?

Cheers,
Sven

Re: [Numpy-discussion] None as missing value

[Sven S. <sve...@gm...>]

Travis Oliphant schrieb:


> 
> You can use a masked array specifically, or use nan's for missing values 
> and just tell Python you want a floating-point array (because it finds 
> the None object it's guessing incorrectly you want an "object" array.
> 
> asarray(x, dtype=float)
> 
> array([[ 1.        ,         nan],
>        [ 2.        ,  3.        ]])
>

Is there anything else besides None which is recognized/converted to
numpy.nan?
Put differently, where can I find documentation about basic nan
definition and handling in numpy? (I have the numpy book which covers
isnan etc., when you have the nans already set up.)

I was also a bit surprised at the following behavior:

>>> a = numpy.asarray([1,1])
>>> a
array([1, 1])
>>> a[0]=numpy.nan
>>> a
array([0, 1])

Is this a bug or intended?

Thanks,
Sven