Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo

Close

#65 dmarrays do not properly convert to masked arrays.

None
open
None
5
2014-03-17
2013-07-09
Dan Welling
No

dmarrays do not convert to np.ma.masked_arrays properly. Specifically, the self.min/max methods fail once converted. This is especially problematic when trying to perform matplotlib contour plots using dmarrays, as the Z-variable handed to the contour function is converted to a masked array to weed out invalid values.

For example

import numpy as np
from spacepy.datamodel import dmarray
x = dmarray(np.arange(10))
x.min() # Works fine.

x2 = np.ma.masked_array(x)
x2.min() #Fails.

x3 = np.ma.masked_array( np.array(x) )
x3.min() # works.

Discussion

  • Steve Morley
    Steve Morley
    2013-07-23

    This is an unforeseen side-effect of making the behaviour of dmarray more like numpy.array. The original problem was that ufuncs like max, min, etc. behave inconsistently between numpy arrays and subclasses of arrays (fix committed in [d4d970]).

    The original problem can be illustrated using the RealisticInfoArray subclass from http://docs.scipy.org/doc/numpy/user/basics.subclassing.html

    RealisticInfoArray([3,4]).min()
    RealisticInfoArray(3) #output is a scalar (zero-dim) array
    np.array([3,4]).min()
    3 #output is a python scalar

    The fix implemented there appears to be responsible for the breakage of numpy's conversion to masked_[subclass], where the masked array actually requires the output of the ufunc to remain as an array-type. I've not found where I can detect the fact that it's using a mask and add special case code for that (yet).

    I'll keep looking for how to fix this, while retaining the fix to min/max/mean/etc.
    The original problem should probably be elevated to numpy though, as it all arises through the fact that subclasses behave differently.

     

    Related

    Commit: [d4d970]

  • As I mentioned on the dev list, MaskedArray itself is a subclass of ndarray, so we can maybe try to steal some ideas from there...