Thread: [Matplotlib-users] Memory problem using pylab.savefig() [bis]

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Dear all,

I posted a message two days ago mentionning memory problems while saving =
a large
number of images, which I guess was not so clear... I've worked a bit on =
it and
I think the questions seem clearer to me now. I have paid particular atte=
ntion
to the threads in the user mailing list dealing with memory problems.

I use a function that generates arrays of typically 500x1380 elements.
For each column, I transform the data and image them using imshow. I need=
n't
display them, all I do is save them. I therefore wrote a function
generateImshow which I show below.

This function allows to produce about 300 out of 500 images and then cras=
hes. I
therefore output the memory usage at each step. I run this from a pyWin s=
hell
on windows XP using a 2.8GHz processor with 512 Kb memory. The result is =
shown
below. The conclusions are the following :

- each cycle leads to an increase of about 6Mb in memory usage, which exp=
lains
why it crashes after some cycles

- if I comment the line pylab.savefig(), then I only get an overall incre=
ase of
about 0.6 Mb, which is a lot, but I can live with this

- obviously, clf() does a good job in freeing memory, but calling the gar=
bage
collector gc.collector does not help, nor does clearing the cached text a=
s
suggested in one of the mails

The problem clearly seems to originate from the savefig operation but I c=
an't
figure out what's going on... anyone has a clue or an idea of how to over=
come
the problem ?

Cheers,

Aur=E9lien

----
def generateImshow(yarray,
                   xscansize,yscansize,
                   xscanstep,yscanstep,
                   outputfilename,
                   initimagesize =3D 10):
    '''Use this function to generate matplotlib images without displaying
them'''

    #
    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step1 ','%.2f' % val,' Mb\n'

    ##normalize file containing intensity correction factors
    #xlist, ylist =3D fileHandling(filename).readChi()
    ##fill in missing data if any
    #ylist.extend([0]*(scanx*scany-len(ylist)))
    #reshape according to scan
    yarray =3D na.reshape(yarray,(yscansize,-1))

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step2 ','%.2f' % val,' Mb'

    #build image via matplotlib
    ximagesize =3D xscansize*xscanstep
    yimagesize =3D yscansize*yscanstep
    xyimageratio =3D float(ximagesize)/yimagesize
    #print xyimageratio
    if xyimageratio > 1: ximagesize,yimagesize =3D
initimagesize,initimagesize*xyimageratio
    else: ximagesize,yimagesize =3D initimagesize*xyimageratio,initimages=
ize

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step3 ','%.2f' % val,' Mb'

    fig1 =3D pylab.figure(figsize=3D(ximagesize,yimagesize),dpi=3D100)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step4 ','%.2f' % val,' Mb'

    fig1.clear()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step5 ','%.2f' % val,' Mb'

    #pylab.title('blahblah')
    im1 =3D pylab.imshow(yarray,
                       origin=3D'lower',
                       #interpolation=3D'nearest',   #i.e. pixel
                       interpolation=3D'bicubic',   #i.e. smooth
                       #vmin=3Dminvalue,
                       #vmax=3Dmaxvalue,
                       cmap =3D pylab.cm.bone,
                       )

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step6 ','%.2f' % val,' Mb'

    colbar1 =3D pylab.colorbar()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step7 ','%.2f' % val,' Mb'

    #pylab.bone()
    pylab.axis('off')

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step8 ','%.2f' % val,' Mb'

    #save figure
    #outputfilename =3D filename[:-4]+'.png'
    pylab.savefig(outputfilename)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step9 ','%.2f' % val,' Mb'

    mpl.text.Text.cached =3D {}

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step10 ','%.2f' % val,' Mb'

    pylab.cla()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step11 ','%.2f' % val,' Mb'

    del im1,colbar1        #doesn't bring anything

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step12 ','%.2f' % val,' Mb'

    pylab.close(fig1)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step13 ','%.2f' % val,' Mb'

    #pylab.close('all')
    gc.collect()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step14 ','%.2f' % val,' Mb'

=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
Result for three steps :

memory used for image - step1  296.57  Mb
memory used for image - step2  296.57  Mb
memory used for image - step3  296.57  Mb
memory used for image - step4  310.75  Mb
memory used for image - step5  310.75  Mb
memory used for image - step6  308.61  Mb
memory used for image - step7  308.71  Mb
memory used for image - step8  308.71  Mb
memory used for image - step9  316.77  Mb
memory used for image - step10  316.77  Mb
memory used for image - step11  316.77  Mb
memory used for image - step12  316.77  Mb
memory used for image - step13  302.92  Mb
memory used for image - step14  302.92  Mb
total time =3D  1.64100003242
done
memory used  302.92  Mb
building file  ...

memory used for image - step1  302.92  Mb
memory used for image - step2  302.92  Mb
memory used for image - step3  302.92  Mb
memory used for image - step4  317.10  Mb
memory used for image - step5  317.10  Mb
memory used for image - step6  314.70  Mb
memory used for image - step7  314.80  Mb
memory used for image - step8  314.80  Mb
memory used for image - step9  322.87  Mb
memory used for image - step10  322.87  Mb
memory used for image - step11  322.87  Mb
memory used for image - step12  322.87  Mb
memory used for image - step13  309.02  Mb
memory used for image - step14  309.02  Mb
total time =3D  1.65599989891
done
memory used  309.02  Mb
building file  ...

memory used for image - step1  309.02  Mb
memory used for image - step2  309.02  Mb
memory used for image - step3  309.02  Mb
memory used for image - step4  323.20  Mb
memory used for image - step5  323.20  Mb
memory used for image - step6  320.79  Mb
memory used for image - step7  320.90  Mb
memory used for image - step8  320.90  Mb
memory used for image - step9  328.97  Mb
memory used for image - step10  328.99  Mb
memory used for image - step11  328.98  Mb
memory used for image - step12  328.98  Mb
memory used for image - step13  315.14  Mb
memory used for image - step14  315.14  Mb
total time =3D  1.875
done

--
Just Aur=E9

Thread: [Matplotlib-users] Memory problem using pylab.savefig() [bis]

matplotlib-users