From: Charlie M. <cw...@gm...> - 2006-02-10 14:22:21
|
Just a quick suggestion. Since you are using pylab and you don't really show a need to recreate a figure everytime, just use "fig1 =3D pylab.gcf()" and don't delete it at the end. Reusing current memory is always going to help prevent leaks. On 2/10/06, aur...@fr... <aur...@fr...> wrote: > Dear all, > > I posted a message two days ago mentionning memory problems while saving = a large > number of images, which I guess was not so clear... I've worked a bit on = it and > I think the questions seem clearer to me now. I have paid particular atte= ntion > to the threads in the user mailing list dealing with memory problems. > > I use a function that generates arrays of typically 500x1380 elements. > For each column, I transform the data and image them using imshow. I need= n't > display them, all I do is save them. I therefore wrote a function > generateImshow which I show below. > > This function allows to produce about 300 out of 500 images and then cras= hes. I > therefore output the memory usage at each step. I run this from a pyWin s= hell > on windows XP using a 2.8GHz processor with 512 Kb memory. The result is = shown > below. The conclusions are the following : > > - each cycle leads to an increase of about 6Mb in memory usage, which exp= lains > why it crashes after some cycles > > - if I comment the line pylab.savefig(), then I only get an overall incre= ase of > about 0.6 Mb, which is a lot, but I can live with this > > - obviously, clf() does a good job in freeing memory, but calling the gar= bage > collector gc.collector does not help, nor does clearing the cached text a= s > suggested in one of the mails > > The problem clearly seems to originate from the savefig operation but I c= an't > figure out what's going on... anyone has a clue or an idea of how to over= come > the problem ? > > Cheers, > > Aur=E9lien > > > ---- > def generateImshow(yarray, > xscansize,yscansize, > xscanstep,yscanstep, > outputfilename, > initimagesize =3D 10): > '''Use this function to generate matplotlib images without displaying > them''' > > # > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step1 ','%.2f' % val,' Mb\n' > > ##normalize file containing intensity correction factors > #xlist, ylist =3D fileHandling(filename).readChi() > ##fill in missing data if any > #ylist.extend([0]*(scanx*scany-len(ylist))) > #reshape according to scan > yarray =3D na.reshape(yarray,(yscansize,-1)) > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step2 ','%.2f' % val,' Mb' > > #build image via matplotlib > ximagesize =3D xscansize*xscanstep > yimagesize =3D yscansize*yscanstep > xyimageratio =3D float(ximagesize)/yimagesize > #print xyimageratio > if xyimageratio > 1: ximagesize,yimagesize =3D > initimagesize,initimagesize*xyimageratio > else: ximagesize,yimagesize =3D initimagesize*xyimageratio,initimages= ize > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step3 ','%.2f' % val,' Mb' > > fig1 =3D pylab.figure(figsize=3D(ximagesize,yimagesize),dpi=3D100) > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step4 ','%.2f' % val,' Mb' > > fig1.clear() > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step5 ','%.2f' % val,' Mb' > > > #pylab.title('blahblah') > im1 =3D pylab.imshow(yarray, > origin=3D'lower', > #interpolation=3D'nearest', #i.e. pixel > interpolation=3D'bicubic', #i.e. smooth > #vmin=3Dminvalue, > #vmax=3Dmaxvalue, > cmap =3D pylab.cm.bone, > ) > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step6 ','%.2f' % val,' Mb' > > colbar1 =3D pylab.colorbar() > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step7 ','%.2f' % val,' Mb' > > #pylab.bone() > pylab.axis('off') > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step8 ','%.2f' % val,' Mb' > > #save figure > #outputfilename =3D filename[:-4]+'.png' > pylab.savefig(outputfilename) > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step9 ','%.2f' % val,' Mb' > > mpl.text.Text.cached =3D {} > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step10 ','%.2f' % val,' Mb' > > pylab.cla() > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step11 ','%.2f' % val,' Mb' > > del im1,colbar1 #doesn't bring anything > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step12 ','%.2f' % val,' Mb' > > pylab.close(fig1) > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step13 ','%.2f' % val,' Mb' > > #pylab.close('all') > gc.collect() > > val =3D float(getMemoryUsage())/1000000 > print 'memory used for image - step14 ','%.2f' % val,' Mb' > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Result for three steps : > > memory used for image - step1 296.57 Mb > memory used for image - step2 296.57 Mb > memory used for image - step3 296.57 Mb > memory used for image - step4 310.75 Mb > memory used for image - step5 310.75 Mb > memory used for image - step6 308.61 Mb > memory used for image - step7 308.71 Mb > memory used for image - step8 308.71 Mb > memory used for image - step9 316.77 Mb > memory used for image - step10 316.77 Mb > memory used for image - step11 316.77 Mb > memory used for image - step12 316.77 Mb > memory used for image - step13 302.92 Mb > memory used for image - step14 302.92 Mb > total time =3D 1.64100003242 > done > memory used 302.92 Mb > building file ... > > memory used for image - step1 302.92 Mb > memory used for image - step2 302.92 Mb > memory used for image - step3 302.92 Mb > memory used for image - step4 317.10 Mb > memory used for image - step5 317.10 Mb > memory used for image - step6 314.70 Mb > memory used for image - step7 314.80 Mb > memory used for image - step8 314.80 Mb > memory used for image - step9 322.87 Mb > memory used for image - step10 322.87 Mb > memory used for image - step11 322.87 Mb > memory used for image - step12 322.87 Mb > memory used for image - step13 309.02 Mb > memory used for image - step14 309.02 Mb > total time =3D 1.65599989891 > done > memory used 309.02 Mb > building file ... > > memory used for image - step1 309.02 Mb > memory used for image - step2 309.02 Mb > memory used for image - step3 309.02 Mb > memory used for image - step4 323.20 Mb > memory used for image - step5 323.20 Mb > memory used for image - step6 320.79 Mb > memory used for image - step7 320.90 Mb > memory used for image - step8 320.90 Mb > memory used for image - step9 328.97 Mb > memory used for image - step10 328.99 Mb > memory used for image - step11 328.98 Mb > memory used for image - step12 328.98 Mb > memory used for image - step13 315.14 Mb > memory used for image - step14 315.14 Mb > total time =3D 1.875 > done > > -- > Just Aur=E9 > > > ------------------------------------------------------- > This SF.net email is sponsored by: Splunk Inc. Do you grep through log fi= les > for problems? Stop! Download the new AJAX search engine that makes > searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! > http://sel.as-us.falkag.net/sel?cmdlnk&kid=103432&bid#0486&dat=121642 > _______________________________________________ > Matplotlib-users mailing list > Mat...@li... > https://lists.sourceforge.net/lists/listinfo/matplotlib-users > |