Menu
▾
▴
matplotlib-users
|
From: Štěpán T. <ste...@se...> - 2013-08-27 07:56:31
|
Hi, I would like to plot multiple overlayed 4096x4096 images in one axes. If I run this code the plot takes 300 MB of memory: import numpy as np import matplotlib.pyplot as plt if __name__ == '__main__': img = np.zeros((4096, 4096)) img[100: 300, 100:1500] = 200 imgplot = plt.imshow(img) plt.show() And it takes additional 300 MB for every image with this size added into plot. Is there any way to reduce memory consumption without need of data resampling? My configuration: Matplotlib 1.2.1 Numpy 1.7.1 Ubuntu 13.04 64 bit Best Stepan |
|
From: Oliver <oli...@gm...> - 2013-08-27 08:14:10
|
You could, before plotting, sum the different image arrays? Depending on whether you are plotting RGB(A) images or greyscale images, you could take the sum of the color channels, or take a weighted average. The method you use here depends strongly on the image type, but it will reduce memory consumption. Just a thought. 2013/8/27 Štěpán Turek <ste...@se...> > Hi, > > I would like to plot multiple overlayed 4096x4096 images in one axes. If I > run this code the plot takes 300 MB of memory: > > import numpy as np > import matplotlib.pyplot as plt > > if __name__ == '__main__': > img = np.zeros((4096, 4096)) > img[100: 300, 100:1500] = 200 > imgplot = plt.imshow(img) > > plt.show() > > And it takes additional 300 MB for every image with this size added into > plot. Is there any way to reduce memory consumption without need of data > resampling? > > My configuration: > Matplotlib 1.2.1 > Numpy 1.7.1 > Ubuntu 13.04 64 bit > > Best > Stepan > > > ------------------------------------------------------------------------------ > Introducing Performance Central, a new site from SourceForge and > AppDynamics. Performance Central is your source for news, insights, > analysis and resources for efficient Application Performance Management. > Visit us today! > http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > |
|
From: Štěpán T. <ste...@se...> - 2013-08-27 08:37:14
|
Hi, " You could, before plotting, sum the different image arrays? Depending on whether you are plotting RGB(A) images or greyscale images, you could take the sum of the color channels, or take a weighted average. " Yes, I will probably merge the images (RGBA) before plotting. I want to create more plots and even with this optimization every plot will take 300 MB... Is there any way how to save some memory? Best Stepan |
|
From: Oliver <oli...@gm...> - 2013-08-27 10:26:42
|
Those numbers actually make a lot of sense. For a 4k by 4k 2D array of 64-bit floats, you're using 128MiB of memory, just to store them. Displaying such an array with mpl would take a copy of that and add some objects for housekeeping (on my machine about 150MB to display one such array together with the housekeeping objects). You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won't get any benefit at all from plotting them together). 2013/8/27 Štěpán Turek <ste...@se...> > Hi, > > > You could, before plotting, sum the different image arrays? Depending on > whether you are plotting RGB(A) images or greyscale images, you could take > the sum of the color channels, or take a weighted average. > > > Yes, I will probably merge the images (RGBA) before plotting. I want to > create more plots and even with this optimization every plot will take 300 > MB... Is there any way how to save some memory? > > > Best > > Stepan > > > |
|
From: Štěpán T. <ste...@se...> - 2013-08-27 10:50:04
|
" "" You could look at whether or not you actually need 64-bit precision. Often times, 8-bit precision per color channel is justifiable, even in grayscale. My advice is to play with the dtype of your array or, as you mentioned, resample. " thanks, this helped me significantly, uint8 precision is enough. " Also, is it needed to keep all images? It sounds to me like your application will become very resource hungry if you're going to be displaying several of these 2D images over each other (and if you don't use transparency, you won' t get any benefit at all from plotting them together). " Yes, I need them all . To avoid it I am thinking about merging them into one image and then plot it. Stepan " " |
|
From: Štěpán T. <ste...@se...> - 2013-08-28 09:19:21
|
Hi Chris, " I've used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image (https://github.com/ChrisBeaumont/mpl-modest-image). " I tried your code for plotting 4kx4k image and it is another significant improvement. Originally it took 300 MB then it was reduced to 190 MB with uint8 type and using your ModestImage class it takes 70-100 MB depending on size of window. That is much better! Best Stepan |
|
From: Chris B. <cbe...@cf...> - 2013-08-27 13:49:20
|
I've been burned by this before as well. MPL stores some intermediate data products (for example, scaled RGB copies) at full resolution, even though the final rendered image is downsampled depending on screen resolution. I've used some hacky tricks to get around this, which mostly involve downsampling the image on the fly based on screen resolution. One such effort is at https://github.com/ChrisBeaumont/mpl-modest-image. If you are loading your arrays from disk, you can also use memory-mapped arrays -- this prevents you from loading all the data into RAM, and further cuts down on the footprint. cheers, chris On Tue, Aug 27, 2013 at 6:49 AM, Štěpán Turek <ste...@se...>wrote: > > You could look at whether or not you actually need 64-bit precision. Often > times, 8-bit precision per color channel is justifiable, even in grayscale. > My advice is to play with the dtype of your array or, as you mentioned, > resample. > > > thanks, this helped me significantly, uint8 precision is enough. > > > > Also, is it needed to keep all images? It sounds to me like your > application will become very resource hungry if you're going to be > displaying several of these 2D images over each other (and if you don't use > transparency, you won't get any benefit at all from plotting them together). > > > Yes, I need them all . > > To avoid it I am thinking about merging them into one image and then plot > it. > > > Stepan > > > > ------------------------------------------------------------------------------ > Introducing Performance Central, a new site from SourceForge and > AppDynamics. Performance Central is your source for news, insights, > analysis and resources for efficient Application Performance Management. > Visit us today! > http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > |
|
From: Michael D. <md...@st...> - 2013-08-27 19:04:37
|
On 08/27/2013 09:49 AM, Chris Beaumont wrote: > I've been burned by this before as well. MPL stores some intermediate > data products (for example, scaled RGB copies) at full resolution, > even though the final rendered image is downsampled depending on > screen resolution. > > I've used some hacky tricks to get around this, which mostly involve > downsampling the image on the fly based on screen resolution. One such > effort is at https://github.com/ChrisBeaumont/mpl-modest-image. It looks like this wouldn't be too hard to include in matplotlib. I don't think we'd want to change the current behavior, because sometimes its tradeoff curve makes sense, but in other cases, the "modest image" approach also makes sense. It's just a matter of coming up with an API to switch between the two behaviors. Pull request? Cheers, Mike > > If you are loading your arrays from disk, you can also use > memory-mapped arrays -- this prevents you from loading all the data > into RAM, and further cuts down on the footprint. > > cheers, > chris > > > On Tue, Aug 27, 2013 at 6:49 AM, S(te(pán Turek > <ste...@se... <mailto:ste...@se...>> wrote: > > > You could look at whether or not you actually need 64-bit > precision. Often times, 8-bit precision per color channel is > justifiable, even in grayscale. My advice is to play with the > dtype of your array or, as you mentioned, resample. > > > thanks, this helped me significantly, uint8 precision is enough. > > Also, is it needed to keep all images? It sounds to me like > your application will become very resource hungry if you're > going to be displaying several of these 2D images over each > other (and if you don't use transparency, you won't get any > benefit at all from plotting them together). > > > Yes, I need them all . > > To avoid it I am thinking about merging them into one image and > then plot it. > > > Stepan > > > ------------------------------------------------------------------------------ > Introducing Performance Central, a new site from SourceForge and > AppDynamics. Performance Central is your source for news, insights, > analysis and resources for efficient Application Performance > Management. > Visit us today! > http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > <mailto:Mat...@li...> > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > > > > > ------------------------------------------------------------------------------ > Introducing Performance Central, a new site from SourceForge and > AppDynamics. Performance Central is your source for news, insights, > analysis and resources for efficient Application Performance Management. > Visit us today! > http://pubads.g.doubleclick.net/gampad/clk?id=48897511&iu=/4140/ostg.clktrk > > > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users |
|
From: Štěpán T. <ste...@se...> - 2013-08-28 07:42:15
|
Hi Martin, "Hi, I knw you asked for memory profiling but I could not resist and did CPU profiling on your testcase. I have attached some screenshots and in words: " thanks for these tips about profiling. Stepan |
