(no subject)

[Jelmer Oosthoek]

Hi Thomas

I’m trying to use Spectral Python to analyse hyperspectral data from the planet Mars. The dataset I’m using for testing is ~600MB and can be found here: http://we.tl/2iPQyeHhi6

I’m using the following code which results in a MemoryError when trying to do envi.save_image:

from spectral import *
img = io.envi.open('frt0000942d_07_if164l_trr3_CAT_scale_trial_p.img.hdr', 'frt0000942d_07_if164l_trr3_CAT_scale_trial_p.img')
pc = principal_components(img)
pc_0999 = pc.reduce(fraction=0.999)
print len(pc_0999.eigenvalues)
img_pc = pc_0999.transform(img)
io.envi.save_image('pc.img.hdr', img_pc)

This is the full error:

Traceback (most recent call last):
File "<interactive input="">", line 1, in <module>
File "build\bdist.win32\egg\spectral\io\envi.py", line 374, in save_image
data = image.load()
File "build\bdist.win32\egg\spectral\io\spyfile.py", line 671, in load
data = self.image.load()
File "build\bdist.win32\egg\spectral\io\spyfile.py", line 229, in load
self.nbands * self.sample_size)
MemoryError

I’ve installed Spectral Python using easy_install on a Windows 7 Professional 64bit computer with Python26 32bit (accompanying ESRI ArcGIS)

Do you maybe have an idea why this MemoryError occurs and how I can solve it?

Kind regards,

Jelmer

[Thomas Boggs]

Hi Jelmer,

You are likely hitting a memory limit associated with 32-bit python (for more details see here or here). The best long-term solution to this problem is to upgrade to 64-bit python but since you are using a bundled version of python, that might not be an option for you.

If you can not upgrade to a 64-bit python, you could try transforming your data pixel by pixel like this:

img_pc = np.empty(tuple(img.shape[:2]) + (len(pc_0999.eigenvalues,)), float)
for i in range(len(img.shape[0])):
    for j in range(len(img.shape[1])):
        img_pc[i, j] = pc_0999.transform(img[i, j])

Then you would save img_pc the same as in your code.

[Jelmer Oosthoek]

Thanks, but I get this error:

img_pc = np.empty(img.shape[:2] + len(pc_0999.eigenvalues,), float)
TypeError: can only concatenate tuple (not "int") to tuple

[Jelmer Oosthoek]

I assume you meant?:

img_pc = np.empty((img.shape[:2][0],img.shape[:2][1],len(pc_0999.eigenvalues)), float)
for i in range(img.shape[0]):
    for j in range(img.shape[1]):
        img_pc[i, j] = pc_0999.transform(img[i, j])

Which gives this error: ComplexWarning: Casting complex values to real discards the imaginary part

But I assume that is OK, as pc.img is written successfully.

[Thomas Boggs]

Yes, the line that created the empty img_pc array was incorrect. I just fixed it and the result should be the same as what you just posted.

[Jelmer Oosthoek]

img_pc = np.empty(tuple(img.shape[:2]) + len(pc_0999.eigenvalues,), float)

Gives error: object of type 'int' has no len()

Would:

img_pc = np.empty((img.shape[:2][0],img.shape[:2][1],len(pc_0999.eigenvalues)), float)

also do the trick?

[Thomas Boggs]

Sorry, I should have checked my typing. I meant to type this:

img_pc = np.empty(tuple(img.shape[:2]) + (len(pc_0999.eigenvalues,)), float)

What you typed should work also. Note that the "[:2]" is not necessary in either of the two locations so you could simplify your version to

img_pc = np.empty((img.shape[0],img.shape[1],len(pc_0999.eigenvalues)), float)

One last note - If you do upgrade to 64-bit python, then you can load the entire image into memory, which will make the PCA calculation an order of magnitude faster if you are on a multicore system.

[Jelmer Oosthoek]

Ok great, thanks. I double checked and I have access to python64 on a multicore machine so I can also run it from there.