numpy-discussion

[Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Webb S. <web...@gm...>]

Could someone recommend a way to average an array along the columns
without propagating the nans and without turning them into some weird
number which bias the result?  I guess I can just keep using an
indexing array for fooArray, but if there is somehting more graceful,
I would love to know.

Boy missing data is a pain in the neck...

Thanks again!

Re: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Eric F. <ef...@ha...>]

Webb Sprague wrote:
> Could someone recommend a way to average an array along the columns
> without propagating the nans and without turning them into some weird
> number which bias the result?  I guess I can just keep using an
> indexing array for fooArray, but if there is somehting more graceful,
> I would love to know.

Something like this:

import numpy as N
ym = N.ma.masked_where(N.isnan(y), y)
yaverage = N.ma.average(ym)

> 
> Boy missing data is a pain in the neck...

It certainly is, but Masked Arrays ease the pain.

Eric

Re: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Sven S. <sve...@gm...>]

Webb Sprague schrieb:
> Could someone recommend a way to average an array along the columns
> without propagating the nans and without turning them into some weird
> number which bias the result?  I guess I can just keep using an
> indexing array for fooArray, but if there is somehting more graceful,
> I would love to know.

You could take advantage of the nan-related functions:


>>> help(numpy.nansum)
Help on function nansum in module numpy.lib.function_base:

nansum(a, axis=-1)
    Sum the array over the given axis, treating NaNs as 0.


> 
> Boy missing data is a pain in the neck...
> 

Btw, do you know what is treated as NaN in numpy when getting the data
from some external source (file, python list etc.), apart from None? I
asked that on this list but it apparently went unnoticed.

Cheers,
Sven

Re: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Sven S. <sve...@gm...>]

Curzio Basso schrieb:
>> Well try it out and see for yourself ;-)
> 
> good point :-)
> 
>> But for sums it doesn't make a difference, right... Note that it's
>> called nan*sum* and not nanwhatever.
> 
> sure, I was still thinking about the first post which was referring to
> the average...
> 
> qrz
> 

Right, having to count the Nans then makes the masked-array solution
more attractive, that's true. So maybe that's a feature request,
complementing the nansum function by a nanaverage?

Re: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Alan G I. <ai...@am...>]

On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: 
> So maybe that's a feature request, complementing the 
> nansum function by a nanaverage? 

This is not an objection; just an observation.
It has always seemed to me that such descriptive
statistics make more sense as class attributes.
In this case, something like a NanDstat class.

fwiw,
Alan Isaac

Re: [Numpy-discussion] numpy.average(fooArray, axis=0): dropping nans from calculation

[Eric F. <ef...@ha...>]

Alan G Isaac wrote:
> On Fri, 14 Jul 2006, Sven Schreiber apparently wrote: 
> 
>>So maybe that's a feature request, complementing the 
>>nansum function by a nanaverage? 
> 
> 
> This is not an objection; just an observation.
> It has always seemed to me that such descriptive
> statistics make more sense as class attributes.
> In this case, something like a NanDstat class.

Attached is something like that, in case someone finds it useful.  It is 
designed to replace something I wrote a long time ago for matlab.  It is 
only very lightly tested, so use with care.

Eric