From: <aur...@fr...> - 2006-02-10 14:08:52
|
Dear all, I posted a message two days ago mentionning memory problems while saving = a large number of images, which I guess was not so clear... I've worked a bit on = it and I think the questions seem clearer to me now. I have paid particular atte= ntion to the threads in the user mailing list dealing with memory problems. I use a function that generates arrays of typically 500x1380 elements. For each column, I transform the data and image them using imshow. I need= n't display them, all I do is save them. I therefore wrote a function generateImshow which I show below. This function allows to produce about 300 out of 500 images and then cras= hes. I therefore output the memory usage at each step. I run this from a pyWin s= hell on windows XP using a 2.8GHz processor with 512 Kb memory. The result is = shown below. The conclusions are the following : - each cycle leads to an increase of about 6Mb in memory usage, which exp= lains why it crashes after some cycles - if I comment the line pylab.savefig(), then I only get an overall incre= ase of about 0.6 Mb, which is a lot, but I can live with this - obviously, clf() does a good job in freeing memory, but calling the gar= bage collector gc.collector does not help, nor does clearing the cached text a= s suggested in one of the mails The problem clearly seems to originate from the savefig operation but I c= an't figure out what's going on... anyone has a clue or an idea of how to over= come the problem ? Cheers, Aur=E9lien ---- def generateImshow(yarray, xscansize,yscansize, xscanstep,yscanstep, outputfilename, initimagesize =3D 10): '''Use this function to generate matplotlib images without displaying them''' # val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step1 ','%.2f' % val,' Mb\n' ##normalize file containing intensity correction factors #xlist, ylist =3D fileHandling(filename).readChi() ##fill in missing data if any #ylist.extend([0]*(scanx*scany-len(ylist))) #reshape according to scan yarray =3D na.reshape(yarray,(yscansize,-1)) val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step2 ','%.2f' % val,' Mb' #build image via matplotlib ximagesize =3D xscansize*xscanstep yimagesize =3D yscansize*yscanstep xyimageratio =3D float(ximagesize)/yimagesize #print xyimageratio if xyimageratio > 1: ximagesize,yimagesize =3D initimagesize,initimagesize*xyimageratio else: ximagesize,yimagesize =3D initimagesize*xyimageratio,initimages= ize val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step3 ','%.2f' % val,' Mb' fig1 =3D pylab.figure(figsize=3D(ximagesize,yimagesize),dpi=3D100) val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step4 ','%.2f' % val,' Mb' fig1.clear() val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step5 ','%.2f' % val,' Mb' #pylab.title('blahblah') im1 =3D pylab.imshow(yarray, origin=3D'lower', #interpolation=3D'nearest', #i.e. pixel interpolation=3D'bicubic', #i.e. smooth #vmin=3Dminvalue, #vmax=3Dmaxvalue, cmap =3D pylab.cm.bone, ) val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step6 ','%.2f' % val,' Mb' colbar1 =3D pylab.colorbar() val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step7 ','%.2f' % val,' Mb' #pylab.bone() pylab.axis('off') val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step8 ','%.2f' % val,' Mb' #save figure #outputfilename =3D filename[:-4]+'.png' pylab.savefig(outputfilename) val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step9 ','%.2f' % val,' Mb' mpl.text.Text.cached =3D {} val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step10 ','%.2f' % val,' Mb' pylab.cla() val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step11 ','%.2f' % val,' Mb' del im1,colbar1 #doesn't bring anything val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step12 ','%.2f' % val,' Mb' pylab.close(fig1) val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step13 ','%.2f' % val,' Mb' #pylab.close('all') gc.collect() val =3D float(getMemoryUsage())/1000000 print 'memory used for image - step14 ','%.2f' % val,' Mb' =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Result for three steps : memory used for image - step1 296.57 Mb memory used for image - step2 296.57 Mb memory used for image - step3 296.57 Mb memory used for image - step4 310.75 Mb memory used for image - step5 310.75 Mb memory used for image - step6 308.61 Mb memory used for image - step7 308.71 Mb memory used for image - step8 308.71 Mb memory used for image - step9 316.77 Mb memory used for image - step10 316.77 Mb memory used for image - step11 316.77 Mb memory used for image - step12 316.77 Mb memory used for image - step13 302.92 Mb memory used for image - step14 302.92 Mb total time =3D 1.64100003242 done memory used 302.92 Mb building file ... memory used for image - step1 302.92 Mb memory used for image - step2 302.92 Mb memory used for image - step3 302.92 Mb memory used for image - step4 317.10 Mb memory used for image - step5 317.10 Mb memory used for image - step6 314.70 Mb memory used for image - step7 314.80 Mb memory used for image - step8 314.80 Mb memory used for image - step9 322.87 Mb memory used for image - step10 322.87 Mb memory used for image - step11 322.87 Mb memory used for image - step12 322.87 Mb memory used for image - step13 309.02 Mb memory used for image - step14 309.02 Mb total time =3D 1.65599989891 done memory used 309.02 Mb building file ... memory used for image - step1 309.02 Mb memory used for image - step2 309.02 Mb memory used for image - step3 309.02 Mb memory used for image - step4 323.20 Mb memory used for image - step5 323.20 Mb memory used for image - step6 320.79 Mb memory used for image - step7 320.90 Mb memory used for image - step8 320.90 Mb memory used for image - step9 328.97 Mb memory used for image - step10 328.99 Mb memory used for image - step11 328.98 Mb memory used for image - step12 328.98 Mb memory used for image - step13 315.14 Mb memory used for image - step14 315.14 Mb total time =3D 1.875 done -- Just Aur=E9 |