From: Thouis (Ray) Jones <thouis@cs...>  20060221 22:04:37

Greetings, I searched the mail archive for information on this question, and didn't find anything. Hopefully this hasn't been addressed elsewhere. I just spent a few hours tracking down a huge performance hit in imshow(). I'm loading some image data via numarray's fromfile(), stacking three imgaes to make an RGB, and showing it with imshow. My image loading code looks like this: def read_dib(filename): f =3D open(filename, "rb") fromfile(f, "UInt8", shape=3D(52, 1)) im =3D fromfile(f, "UInt16", shape=3D(512, 512)) if sys.byteorder =3D=3D "big": =09im.byteswap() return im.astype('Float') / (2**12  1) I load three images, and put them into a NxMx3 float numarray, and pass that to imshow. The performance drop is at this line in image.py: im =3D _image.fromarray(x, 0) in AxesImage.make_image() I eventually tracked it down to x's type being <class 'numarray.ma.MA.MaskedArray'>, which gets passed off to numarray's NA_InputArray function, which chokes and calls setArrayFromSequence (i.e., it's not using the fact that the input data is a numarray, and is instead iterating over each element). This is taking around 30 seconds to convert a 512x512x3 image on a new Intel iMac. I sped it up significantly by using this line instead: im =3D _image.fromarray(numerix.ma.filled(x,0), 0) Not that that's a good idea (but it works for me for now). There is no invalid data in my image. I'm not sure why it's ever being turned into a masked array. I'm using matplotlib 0.86.2 and numarray1.5.1, on OS X 10.3.5 on an Intel = iMac. Any advice would be appreciated. Thanks, Ray Jones 
From: Eric Firing <efiring@ha...>  20060222 08:31:14

Ray, You tripped over a bug in cm.py: when I added masked array support, I put the "x = ma.asarray(x)" too early in the to_rgba() method of ScalarMappable. It is fixed now in CVS. Thanks for finding the problem. If you want to try the fix in your version, in place of your workaround, here is the revised method: def to_rgba(self, x, alpha=1.0): # assume normalized rgb, rgba if len(x.shape)>2: return x x = ma.asarray(x) x = self.norm(x) x = self.cmap(x, alpha) return x Eric Thouis (Ray) Jones wrote: > Greetings, > > I searched the mail archive for information on this question, and > didn't find anything. Hopefully this hasn't been addressed elsewhere. > > I just spent a few hours tracking down a huge performance hit in > imshow(). I'm loading some image data via numarray's fromfile(), > stacking three imgaes to make an RGB, and showing it with imshow. > > My image loading code looks like this: > > def read_dib(filename): > f = open(filename, "rb") > fromfile(f, "UInt8", shape=(52, 1)) > im = fromfile(f, "UInt16", shape=(512, 512)) > if sys.byteorder == "big": > im.byteswap() > return im.astype('Float') / (2**12  1) > > I load three images, and put them into a NxMx3 float numarray, and > pass that to imshow. > > The performance drop is at this line in image.py: > im = _image.fromarray(x, 0) > > in AxesImage.make_image() > > I eventually tracked it down to x's type being <class > 'numarray.ma.MA.MaskedArray'>, which gets passed off to numarray's > NA_InputArray function, which chokes and calls setArrayFromSequence > (i.e., it's not using the fact that the input data is a numarray, and > is instead iterating over each element). This is taking around 30 > seconds to convert a 512x512x3 image on a new Intel iMac. > > I sped it up significantly by using this line instead: > > im = _image.fromarray(numerix.ma.filled(x,0), 0) > > Not that that's a good idea (but it works for me for now). > > There is no invalid data in my image. I'm not sure why it's ever > being turned into a masked array. > > I'm using matplotlib 0.86.2 and numarray1.5.1, on OS X 10.3.5 on an Intel iMac. > > Any advice would be appreciated. > > Thanks, > Ray Jones > > >  > This SF.net email is sponsored by: Splunk Inc. Do you grep through log files > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.asus.falkag.net/sel?cmd=k&kid3432&bid#0486&dat1642 > _______________________________________________ > Matplotlibusers mailing list > Matplotlibusers@... > https://lists.sourceforge.net/lists/listinfo/matplotlibusers 
From: Jeff Whitaker <jswhit@fa...>  20060222 17:27:40

Eric Firing wrote: > Ray, > > You tripped over a bug in cm.py: when I added masked array support, I > put the "x = ma.asarray(x)" too early in the to_rgba() method of > ScalarMappable. It is fixed now in CVS. Thanks for finding the problem. > > If you want to try the fix in your version, in place of your > workaround, here is the revised method: > > def to_rgba(self, x, alpha=1.0): > # assume normalized rgb, rgba > if len(x.shape)>2: return x > x = ma.asarray(x) > x = self.norm(x) > x = self.cmap(x, alpha) > return x > > Eric > Hi Eric: This breaks some of the basemap demos which do this CS = m.contour(x,y,hgt,15,linewidths=0.5,colors='k') to draw solid black contour lines. I now get File "/sw/lib/python2.4/sitepackages/matplotlib/cm.py", line 52, in to_rgba if len(x.shape)>2: return x AttributeError: 'list' object has no attribute 'shape' It appears that colors now has to be an array? Jeff  Jeffrey S. Whitaker Phone : (303)4976313 Meteorologist FAX : (303)4976449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@... 325 Broadway Office : Skaggs Research Cntr 1D124 Boulder, CO, USA 803033328 Web : http://tinyurl.com/5telg 
From: Eric Firing <efiring@ha...>  20060222 18:16:11

Jeff, I really blew it on that oneit looked so simple! OK, I think this version is correct; it will land in CVS in a few minutes. def to_rgba(self, x, alpha=1.0): '''Return a normalized rgba array corresponding to x. If x is already an rgb or rgba array, return it unchanged. ''' if hasattr(x, 'shape') and len(x.shape)>2: return x x = ma.asarray(x) x = self.norm(x) x = self.cmap(x, alpha) return x Eric Jeff Whitaker wrote: > Eric Firing wrote: > >> Ray, >> >> You tripped over a bug in cm.py: when I added masked array support, I >> put the "x = ma.asarray(x)" too early in the to_rgba() method of >> ScalarMappable. It is fixed now in CVS. Thanks for finding the problem. >> >> If you want to try the fix in your version, in place of your >> workaround, here is the revised method: >> >> def to_rgba(self, x, alpha=1.0): >> # assume normalized rgb, rgba >> if len(x.shape)>2: return x >> x = ma.asarray(x) >> x = self.norm(x) >> x = self.cmap(x, alpha) >> return x >> >> Eric >> > > > Hi Eric: This breaks some of the basemap demos which do this > > CS = m.contour(x,y,hgt,15,linewidths=0.5,colors='k') > > to draw solid black contour lines. I now get > > File "/sw/lib/python2.4/sitepackages/matplotlib/cm.py", line 52, in > to_rgba > if len(x.shape)>2: return x > AttributeError: 'list' object has no attribute 'shape' > > It appears that colors now has to be an array? > > Jeff > 
From: Jeff Whitaker <jswhit@fa...>  20060222 20:11:53

Eric Firing wrote: > Jeff, > > I really blew it on that oneit looked so simple! OK, I think this > version is correct; it will land in CVS in a few minutes. > > def to_rgba(self, x, alpha=1.0): > '''Return a normalized rgba array corresponding to x. > If x is already an rgb or rgba array, return it unchanged. > ''' > if hasattr(x, 'shape') and len(x.shape)>2: return x > x = ma.asarray(x) > x = self.norm(x) > x = self.cmap(x, alpha) > return x > Eric > Eric: Thanks for the quick fix  that seems to have done the job. Jeff  Jeffrey S. Whitaker Phone : (303)4976313 Meteorologist FAX : (303)4976449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@... 325 Broadway Office : Skaggs Research Cntr 1D124 Boulder, CO, USA 803033328 Web : http://tinyurl.com/5telg 
From: Andrew Jaffe <a.jaffe@gm...>  20060223 11:20:12

Jeff Whitaker wrote: > Eric Firing wrote: >> Jeff, >> I really blew it on that oneit looked so simple! OK, I think this >> version is correct; it will land in CVS in a few minutes. >> >> def to_rgba(self, x, alpha=1.0): >> [[[ code removed ]]] >> > Eric: Thanks for the quick fix  that seems to have done the job. > Jeff I don't know if it's related, but as of the latest version, all of my imshow() commands show a single color, no structure at all. 
From: Eric Firing <efiring@ha...>  20060223 17:31:41

Andrew, Unfortunately, the 0.87 release was packed up immediately after I made the first of two changes to cm.py, so it contains the bug referred to below. Most likely that is what is wrecking your imshow plots. Please try substituting the revised function in cm.py, and let us know if that solves the problem. def to_rgba(self, x, alpha=1.0): '''Return a normalized rgba array corresponding to x. If x is already an rgb or rgba array, return it unchanged. ''' if hasattr(x, 'shape') and len(x.shape)>2: return x x = ma.asarray(x) x = self.norm(x) x = self.cmap(x, alpha) return x Eric Andrew Jaffe wrote: > Jeff Whitaker wrote: > >> Eric Firing wrote: >> >>> Jeff, >>> I really blew it on that oneit looked so simple! OK, I think this >>> version is correct; it will land in CVS in a few minutes. >>> >>> def to_rgba(self, x, alpha=1.0): > > >> [[[ code removed ]]] > >> > >> Eric: Thanks for the quick fix  that seems to have done the job. >> Jeff > > > I don't know if it's related, but as of the latest version, all of my > imshow() commands show a single color, no structure at all. 
From: Andrew Jaffe <a.jaffe@gm...>  20060223 22:00:46

Eric (etc) Sorry, no dice... in fact I was already using the CVS version so this was the to_rgba() method that failed originally! Any other possibilities? Could the problem be just a coincidental error elsewhere? Andrew On 23 Feb 2006, at 17:31, Eric Firing wrote: > Andrew, > > Unfortunately, the 0.87 release was packed up immediately after I > made the first of two changes to cm.py, so it contains the bug > referred to below. Most likely that is what is wrecking your > imshow plots. Please try substituting the revised function in > cm.py, and let us know if that solves the problem. > > def to_rgba(self, x, alpha=1.0): > '''Return a normalized rgba array corresponding to x. > If x is already an rgb or rgba array, return it unchanged. > ''' > if hasattr(x, 'shape') and len(x.shape)>2: return x > x = ma.asarray(x) > x = self.norm(x) > x = self.cmap(x, alpha) > return x > > Eric > > Andrew Jaffe wrote: >> Jeff Whitaker wrote: >>> Eric Firing wrote: >>> >>>> Jeff, >>>> I really blew it on that oneit looked so simple! OK, I think >>>> this version is correct; it will land in CVS in a few minutes. >>>> >>>> def to_rgba(self, x, alpha=1.0): >> >> [[[ code removed ]]] >> >> >>> Eric: Thanks for the quick fix  that seems to have done the job. >>> Jeff >> I don't know if it's related, but as of the latest version, all of >> my imshow() commands show a single color, no structure at all. ______________________________________________________________________ Andrew Jaffe a.jaffe@... Astrophysics Group +44 207 5947526 Blackett Laboratory, Room 1013 FAX 7541 Imperial College, Prince Consort Road London SW7 2AZ ENGLAND http://astro.imperial.ac.uk/~jaffe 
From: Jeff Whitaker <jswhit@fa...>  20060223 22:29:57

Andrew Jaffe wrote: > Eric (etc) > > Sorry, no dice... in fact I was already using the CVS version so this > was the to_rgba() method that failed originally! > > Any other possibilities? Could the problem be just a coincidental > error elsewhere? > > Andrew > > Andrew and Eric: I apologize, this latest glitch was all my fault. I had made a modification to colors.py to workaround an apparenty bug in numpy masked arrays, and that modification was wrong. It's now fixed in CVS, revision 1.26 Index: colors.py =================================================================== RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v retrieving revision 1.25 diff r1.25 colors.py 659c659 < result = (vmaxvmin)/(valvmin) < this is wrong  > result = (valvmin)/(vmaxvmin) < this is right Originally (in 0.87) there was result = (1.0/(vmaxvmin))*(valvmin) which raised an exception when val was a numpy masked array. Sorry for the screwup. Jeff  Jeffrey S. Whitaker Phone : (303)4976313 Meteorologist FAX : (303)4976449 NOAA/OAR/PSD R/PSD1 Email : Jeffrey.S.Whitaker@... 325 Broadway Office : Skaggs Research Cntr 1D124 Boulder, CO, USA 803033328 Web : http://tinyurl.com/5telg 
From: Andrew Jaffe <a.jaffe@gm...>  20060223 23:02:50

Hi Thanks for the help. A comment below... > Andrew Jaffe wrote: >> Eric (etc) >> >> Sorry, no dice... in fact I was already using the CVS version so >> this was the to_rgba() method that failed originally! >> >> Any other possibilities? Could the problem be just a coincidental >> error elsewhere? >> >> Andrew >> >> > > Andrew and Eric: I apologize, this latest glitch was all my > fault. I had made a modification to colors.py to workaround an > apparenty bug in numpy masked arrays, and that modification was > wrong. It's now fixed in CVS, revision 1.26 > > Index: colors.py > =================================================================== > RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v > retrieving revision 1.25 > diff r1.25 colors.py > 659c659 > < result = (vmaxvmin)/(valvmin) < this is wrong >  > > result = (valvmin)/(vmaxvmin) < this is right > > Originally (in 0.87) there was > > result = (1.0/(vmaxvmin))*(valvmin) > > which raised an exception when val was a numpy masked array. However, I expect the original line was like this in case vmax, vmin and val are all integers, in which case without 'truedivision' (or 'from __future__ import division'), result will use integer division which probably isn't desired. So this may need to be fixed to force floatingpoint division. Not sure the robust way to do this with possiblymasked arrays, etc. Andrew 
From: Jeff Whitaker <jswhit@fa...>  20060224 02:38:26

Andrew Jaffe wrote: > Hi > > Thanks for the help. A comment below... > >> Andrew Jaffe wrote: >>> Eric (etc) >>> >>> Sorry, no dice... in fact I was already using the CVS version so >>> this was the to_rgba() method that failed originally! >>> >>> Any other possibilities? Could the problem be just a coincidental >>> error elsewhere? >>> >>> Andrew >>> >>> >> >> Andrew and Eric: I apologize, this latest glitch was all my fault. >> I had made a modification to colors.py to workaround an apparenty bug >> in numpy masked arrays, and that modification was wrong. It's now >> fixed in CVS, revision 1.26 >> >> Index: colors.py >> =================================================================== >> RCS file: /cvsroot/matplotlib/matplotlib/lib/matplotlib/colors.py,v >> retrieving revision 1.25 >> diff r1.25 colors.py >> 659c659 >> < result = (vmaxvmin)/(valvmin) < this is wrong >>  >> > result = (valvmin)/(vmaxvmin) < this is right >> >> Originally (in 0.87) there was >> >> result = (1.0/(vmaxvmin))*(valvmin) >> >> which raised an exception when val was a numpy masked array. > > However, I expect the original line was like this in case vmax, vmin > and val are all integers, in which case without 'truedivision' (or > 'from __future__ import division'), result will use integer division > which probably isn't desired. > > So this may need to be fixed to force floatingpoint division. Not > sure the robust way to do this with possiblymasked arrays, etc. > > Andrew > > > Andrew: (valvmin)/float(vmaxvmin) should be equivalent to the original, since I believe vmin and vmax are always scalars. BTW: you can trigger the numpy.ma/imshow problem by uncommenting the 'colorbar(im)' line in the image_masked.py example. With the original result = (1.0/(vmaxvmin))*(valvmin) line in colors.py, it will throw an exception with numpy 0.9.5, but result = (valvmin)/float(vmaxvmin) works. Jeff 