From: sprobst <Ste...@gm...> - 2011-01-14 19:40:50
|
Hi all, I tried to plot parts of a large 3D array with each 4 float64 entries. Loading the array with numpy.fromfile and performing a type conversion afterwards ends up in a MemoryError. The following code reproduces the error: *********************************** import gc from os import path from numpy import zeros, empty, float32, float64, fromfile # With the imports of the next three lines a memory error occurs #import matplotlib #matplotlib.use("AGG") #import matplotlib.pylab as plt _filename = "mt.dat" if (path.exists(_filename)==False): print "Write file ..." _mtf = file(_filename, "wb") _mtd = zeros( (300,300,300,4),dtype=float64) _mtd.tofile(_mtf) _mtf.close() _mtd = empty((0)) gc.collect(); print "Try to read file ..." _mtf = file(_filename, "rb") # The memory error occurs with the type conversion to float 64, # but only if mathplotlib is imported! _mtd = fromfile(_mtf,float64).astype(float32) _mtf.close() print "Successful read:",_mtd.shape # Here some plot stuff would be :) *********************************** If the import of matplotlib is not included, no error occures otherwise it will not work. Perhaps anybody can help me (I tested it only on a Windows XP 32Bit). Best regards and thank you Stefan -- View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30674798.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
From: Christoph G. <cg...@uc...> - 2011-01-14 20:02:26
|
Works for me on Windows 7 64 bit with 32 bit Python. I believe you simply run out of memory. On a 32 bit Windows OS, Python can only use 2 GB, which it has to share with other all processes. zeros((300,300,300,4),dtype=float64) requires ~820MB of contiguous memory, which might not be available even if you have >1 GB free RAM. Better use a 64 bit OS and Python. Christoph On 1/14/2011 11:40 AM, sprobst wrote: > > Hi all, > > I tried to plot parts of a large 3D array with each 4 float64 entries. > Loading the array with numpy.fromfile and performing a type conversion > afterwards ends up in a MemoryError. > > The following code reproduces the error: > *********************************** > import gc > > from os import path > from numpy import zeros, empty, float32, float64, fromfile > > # With the imports of the next three lines a memory error occurs > #import matplotlib > #matplotlib.use("AGG") > #import matplotlib.pylab as plt > > _filename = "mt.dat" > > if (path.exists(_filename)==False): > print "Write file ..." > _mtf = file(_filename, "wb") > _mtd = zeros( (300,300,300,4),dtype=float64) > _mtd.tofile(_mtf) > _mtf.close() > > _mtd = empty((0)) > gc.collect(); > > print "Try to read file ..." > > _mtf = file(_filename, "rb") > # The memory error occurs with the type conversion to float 64, > # but only if mathplotlib is imported! > _mtd = fromfile(_mtf,float64).astype(float32) > _mtf.close() > > print "Successful read:",_mtd.shape > > # Here some plot stuff would be :) > *********************************** > > If the import of matplotlib is not included, no error occures otherwise it > will not work. Perhaps anybody can help me (I tested it only on a Windows XP > 32Bit). > > Best regards and thank you > Stefan |
From: sprobst <Ste...@gm...> - 2011-01-14 20:17:35
|
Dear Christoph, It is not a memory problem, because before the programm starts I have more than 1.3 GB free memory left. And on the other hand it depends on the import line! There seems to be some interference of the packages. My Python version is 2.6 (needed by another programm), perhaps that also makes the difference. But thank you for the quick answer! Stefan -- View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30675089.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
From: Christoph G. <cg...@uc...> - 2011-01-14 20:50:16
|
The zeros(...) statement needs a contiguous block of > 800 MB RAM and the conversion another contiguous block of > 400 MB. Memory allocated during the matplotlib import statement could easily fragment the available memory such that no 800 + 400 MB blocks exist. Try monitoring your memory usage and import matplotlib after allocating/converting the array. Anyway, you'll be better off using a 64 bit system if you deal with such amount of data. Christoph On 1/14/2011 12:17 PM, sprobst wrote: > > Dear Christoph, > > It is not a memory problem, because before the programm starts I have more > than 1.3 GB free memory left. And on the other hand it depends on the import > line! There seems to be some interference of the packages. My Python version > is 2.6 (needed by another programm), perhaps that also makes the difference. > > But thank you for the quick answer! > Stefan > |
From: sprobst <Ste...@gm...> - 2011-01-15 07:56:25
|
Problem solved ... even as orginally C++ or Java programmer it is strange, but a interpreted language makes it possible to obtain with an import at a different position a complety different behaviour during run-time. I just putted the import of mathplotlib and pylab after the file reading and after the type conversion and it works. :jumping: Comment to just switch to another system with 64bit. Sometimes if your are not admin of a machine, than it is hard to change the system ;-) Thank you anyhow, based on your comments I found the solution. Do you perhaps know if there is something like a memdefrag :-/? Have a nice weekend Stefan -- View this message in context: http://old.nabble.com/MemoryError-with-import-matplotlib-tp30674798p30676114.html Sent from the matplotlib - users mailing list archive at Nabble.com. |
From: Benjamin R. <ben...@ou...> - 2011-01-14 20:03:43
|
On Fri, Jan 14, 2011 at 1:40 PM, sprobst <Ste...@gm...> wrote: > > Hi all, > > I tried to plot parts of a large 3D array with each 4 float64 entries. > Loading the array with numpy.fromfile and performing a type conversion > afterwards ends up in a MemoryError. > > The following code reproduces the error: > *********************************** > import gc > > from os import path > from numpy import zeros, empty, float32, float64, fromfile > > # With the imports of the next three lines a memory error occurs > #import matplotlib > #matplotlib.use("AGG") > #import matplotlib.pylab as plt > > _filename = "mt.dat" > > if (path.exists(_filename)==False): > print "Write file ..." > _mtf = file(_filename, "wb") > _mtd = zeros( (300,300,300,4),dtype=float64) > _mtd.tofile(_mtf) > _mtf.close() > > _mtd = empty((0)) > gc.collect(); > > print "Try to read file ..." > > _mtf = file(_filename, "rb") > # The memory error occurs with the type conversion to float 64, > # but only if mathplotlib is imported! > _mtd = fromfile(_mtf,float64).astype(float32) > _mtf.close() > > print "Successful read:",_mtd.shape > > # Here some plot stuff would be :) > *********************************** > > If the import of matplotlib is not included, no error occures otherwise it > will not work. Perhaps anybody can help me (I tested it only on a Windows > XP > 32Bit). > > Best regards and thank you > Stefan > Stefan, I don't know if it would make a difference, but I see that you are importing matplotlib.pylab as plt. This is not a typical way of importing matplotlib. Try instead: import matplotlib.pyplot as plt Ben Root |