Re: [Matplotlib-users] Memory problem using pylab.savefig() [bis]

[<aur...@fr...>]

Dear John,

I finnaly found a solution that works following the suggestion from Charl=
ie Moad
(many thanks !).

Before this, I tried what you suggested (explicitly naming the figure). T=
he
output for 3 cycles is shown bellow in result1. It doesn't change much in=
 the
story, the memory still increases by about 6Mb. I also show the result wh=
en I
uncomment the savefig line (result2) which also shows an increase by abou=
t 0.6
Mb.

If instead of actually closing and opening a figure each time I use gcf (=
without
closing the figure), then, if works fine (result3). Unlike for the first =
two
cases, the memory is freed using the gc.collector (instead of close).

The final code is at the bottom. I still don't uderstand why it doesn't w=
ork by
the first method, but at least it works...

Many thanks,

Cheers,

Aur=E9lien


--------------
RESULT 1:

memory used  331.55  Mb
Building individual images...
building file  ...
memory used for image - step1  331.55  Mb
memory used for image - step2  331.55  Mb
memory used for image - step3  331.55  Mb
memory used for image - step4  331.55  Mb	-> created fig
memory used for image - step5  331.55  Mb	-> cleared fig
memory used for image - step6  329.11  Mb	-> built image - imshow
memory used for image - step7  329.20  Mb	-> added colorbar
memory used for image - step8  329.20  Mb	-> turned axis off
memory used for image - step9  336.74  Mb	-> saved fig
memory used for image - step10  336.74  Mb	-> cleared cached text
memory used for image - step11  336.74  Mb	-> cleared image (axes)
memory used for image - step12  336.74  Mb
memory used for image - step13  328.31  Mb	-> closed figure
memory used for image - step14  328.31  Mb	-> called garbage collector
total time =3D  1.84299993515
done
memory used  328.31  Mb
building file  ...
memory used for image - step1  328.31  Mb
memory used for image - step2  328.31  Mb
memory used for image - step3  328.31  Mb
memory used for image - step4  342.26  Mb	-> created fig
memory used for image - step5  342.26  Mb	-> cleared fig
memory used for image - step6  339.84  Mb	-> built image - imshow
memory used for image - step7  339.94  Mb	-> added colorbar
memory used for image - step8  339.94  Mb	-> turned axis off
memory used for image - step9  347.50  Mb	-> saved fig
memory used for image - step10  347.50  Mb	-> cleared cached text
memory used for image - step11  347.50  Mb	-> cleared image (axes)
memory used for image - step12  347.50  Mb
memory used for image - step13  333.66  Mb	-> closed figure
memory used for image - step14  333.66  Mb	-> called garbage collector
total time =3D  1.93700003624
done
memory used  333.66  Mb
building file  ...
memory used for image - step1  333.66  Mb
memory used for image - step2  333.66  Mb
memory used for image - step3  333.66  Mb
memory used for image - step4  347.81  Mb	-> created fig
memory used for image - step5  347.81  Mb	-> cleared fig
memory used for image - step6  345.40  Mb	-> built image - imshow
memory used for image - step7  345.51  Mb	-> added colorbar
memory used for image - step8  345.51  Mb	-> turned axis off
memory used for image - step9  353.57  Mb	-> saved fig
memory used for image - step10  353.57  Mb	-> cleared cached text
memory used for image - step11  353.57  Mb	-> cleared image (axes)
memory used for image - step12  353.57  Mb
memory used for image - step13  339.73  Mb	-> closed figure
memory used for image - step14  339.73  Mb	-> called garbage collector
total time =3D  1.67200016975
done

--
RESULT 2:

memory used  345.79  Mb
Building individual images...
building file=20
I:/EggShell/ESRF-sc1579-201104/OriginalDataPolarTransform/egg4_8_0500_pol=
ar_mask.png
 ...
memory used for image - step1  345.79  Mb
memory used for image - step2  345.79  Mb
memory used for image - step3  345.79  Mb
memory used for image - step4  359.92  Mb	-> created fig
memory used for image - step5  359.92  Mb	-> cleared fig
memory used for image - step6  357.51  Mb	-> built image - imshow
memory used for image - step7  357.61  Mb	-> added colorbar
memory used for image - step8  357.61  Mb	-> turned axis off
memory used for image - step9  357.61  Mb	-> saved fig - DISABLED
memory used for image - step10  357.61  Mb	-> cleared cached text
memory used for image - step11  357.61  Mb	-> cleared image (axes)
memory used for image - step12  357.61  Mb
memory used for image - step13  343.77  Mb	-> closed figure
memory used for image - step14  343.77  Mb	-> called garbage collector
total time =3D  1.25
done
memory used  343.77  Mb
building file=20
I:/EggShell/ESRF-sc1579-201104/OriginalDataPolarTransform/egg4_8_0499_pol=
ar_mask.png
 ...
memory used for image - step1  343.77  Mb
memory used for image - step2  343.77  Mb
memory used for image - step3  344.02  Mb
memory used for image - step4  358.22  Mb	-> created fig
memory used for image - step5  358.22  Mb	-> cleared fig
memory used for image - step6  358.28  Mb	-> built image - imshow
memory used for image - step7  358.38  Mb	-> added colorbar
memory used for image - step8  358.38  Mb	-> turned axis off
memory used for image - step9  358.38  Mb	-> saved fig - DISABLED
memory used for image - step10  358.38  Mb	-> cleared cached text
memory used for image - step11  358.38  Mb	-> cleared image (axes)
memory used for image - step12  358.38  Mb
memory used for image - step13  344.54  Mb	-> closed figure
memory used for image - step14  344.54  Mb	-> called garbage collector
total time =3D  0.625
done
memory used  344.54  Mb
building file=20
I:/EggShell/ESRF-sc1579-201104/OriginalDataPolarTransform/egg4_8_0498_pol=
ar_mask.png
 ...
memory used for image - step1  344.54  Mb
memory used for image - step2  344.54  Mb
memory used for image - step3  344.54  Mb
memory used for image - step4  358.74  Mb	-> created fig
memory used for image - step5  358.75  Mb	-> cleared fig
memory used for image - step6  358.81  Mb	-> built image - imshow
memory used for image - step7  358.91  Mb	-> added colorbar
memory used for image - step8  358.91  Mb	-> turned axis off
memory used for image - step9  358.91  Mb	-> saved fig - DISABLED
memory used for image - step10  358.91  Mb	-> cleared cached text
memory used for image - step11  358.91  Mb	-> cleared image (axes)
memory used for image - step12  358.91  Mb
memory used for image - step13  345.06  Mb	-> closed figure
memory used for image - step14  345.06  Mb	-> called garbage collector
total time =3D  0.640000104904
done

--
RESULT 3:

memory used  325.69  Mb
Building individual images...
building file ...
memory used for image - step1  325.69  Mb
memory used for image - step2  325.41  Mb
memory used for image - step3  325.41  Mb
memory used for image - step4  325.41  Mb	-> created fig - gcf
memory used for image - step5  325.41  Mb	-> cleared fig
memory used for image - step6  322.70  Mb	-> built image - imshow
memory used for image - step7  322.61  Mb	-> added colorbar
memory used for image - step8  322.61  Mb	-> turned axis off
memory used for image - step9  330.62  Mb	-> saved fig
memory used for image - step10  330.62  Mb	-> cleared cached text
memory used for image - step11  330.62  Mb	-> cleared image (axes)
memory used for image - step12  330.62  Mb
memory used for image - step13  330.62  Mb	-> closed figure - DISABLED
memory used for image - step14  325.12  Mb	-> called garbage collector
total time =3D  1.28099989891
done
memory used  325.12  Mb
building file  ...
memory used for image - step1  325.12  Mb
memory used for image - step2  325.12  Mb
memory used for image - step3  325.12  Mb
memory used for image - step4  325.12  Mb	-> created fig - gcf
memory used for image - step5  325.12  Mb	-> cleared fig
memory used for image - step6  322.65  Mb	-> built image - imshow
memory used for image - step7  322.65  Mb	-> added colorbar
memory used for image - step8  322.65  Mb	-> turned axis off
memory used for image - step9  330.63  Mb	-> saved fig
memory used for image - step10  330.63  Mb	-> cleared cached text
memory used for image - step11  330.63  Mb	-> cleared image (axes)
memory used for image - step12  330.63  Mb
memory used for image - step13  330.63  Mb	-> closed figure - DISABLED
memory used for image - step14  325.13  Mb	-> called garbage collector
total time =3D  1.31299996376
done
memory used  325.13  Mb
building file ...
memory used for image - step1  325.13  Mb
memory used for image - step2  325.13  Mb
memory used for image - step3  325.13  Mb
memory used for image - step4  325.13  Mb	-> created fig - gcf
memory used for image - step5  325.13  Mb	-> cleared fig
memory used for image - step6  322.62  Mb	-> built image - imshow
memory used for image - step7  322.63  Mb	-> added colorbar
memory used for image - step8  322.63  Mb	-> turned axis off
memory used for image - step9  330.64  Mb	-> saved fig
memory used for image - step10  330.64  Mb	-> cleared cached text
memory used for image - step11  330.64  Mb	-> cleared image (axes)
memory used for image - step12  330.64  Mb
memory used for image - step13  330.64  Mb	-> closed figure - DISABLED
memory used for image - step14  325.09  Mb	-> called garbage collector
total time =3D  1.31299996376
done

--

def generateImshow(yarray,
                   xscansize,yscansize,
                   xscanstep,yscanstep,
                   outputfilename,
                   initimagesize =3D 10):
    '''Use this function to generate matplotlib images without displaying
them'''

    #
    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step1 ','%.2f' % val,' Mb'

    #reshape according to scan
    yarray =3D na.reshape(yarray,(yscansize,-1))

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step2 ','%.2f' % val,' Mb'

    #build image via matplotlib
    ximagesize =3D xscansize*xscanstep
    yimagesize =3D yscansize*yscanstep
    xyimageratio =3D float(ximagesize)/yimagesize
    #print xyimageratio
    if xyimageratio > 1: ximagesize,yimagesize =3D
initimagesize,initimagesize*xyimageratio
    else: ximagesize,yimagesize =3D initimagesize*xyimageratio,initimages=
ize

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step3 ','%.2f' % val,' Mb'

    #fig1 =3D pylab.figure(figsize=3D(ximagesize,yimagesize),dpi=3D100)
    #fig1 =3D pylab.figure(1,figsize=3D(ximagesize,yimagesize),dpi=3D100)
    fig1 =3D pylab.gcf()
    fig1.set_figsize_inches((ximagesize,yimagesize))
    fig1.set_dpi(100)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step4 ','%.2f' % val,' Mb'

    fig1.clear()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step5 ','%.2f' % val,' Mb'


    #pylab.title('blahblah')
    im1 =3D pylab.imshow(yarray,
                       origin=3D'lower',
                       #interpolation=3D'nearest',   #i.e. pixel
                       interpolation=3D'bicubic',   #i.e. smooth
                       #vmin=3Dminvalue,
                       #vmax=3Dmaxvalue,
                       cmap =3D pylab.cm.bone,
                       )

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step6 ','%.2f' % val,' Mb'

    colbar1 =3D pylab.colorbar()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step7 ','%.2f' % val,' Mb'

    #pylab.bone()
    pylab.axis('off')

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step8 ','%.2f' % val,' Mb'

    #save figure
    pylab.savefig(outputfilename)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step9 ','%.2f' % val,' Mb'

    mpl.text.Text.cached =3D {}

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step10 ','%.2f' % val,' Mb'

    pylab.cla()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step11 ','%.2f' % val,' Mb'

    del im1,colbar1        #doesn't bring anything

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step12 ','%.2f' % val,' Mb'

    #pylab.close(fig1)
    #pylab.close(1)

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step13 ','%.2f' % val,' Mb'

    #pylab.close('all')
    gc.collect()

    val =3D float(getMemoryUsage())/1000000
    print 'memory used for image - step14 ','%.2f' % val,' Mb'
-------------------------



Selon John Hunter <jdh...@ac...>:

> >>>>> "aurelien" =3D=3D aurelien gourrier <aur...@fr...> w=
rites:
>
>     aurelien> Dear all, I posted a message two days ago mentionning
>     aurelien> memory problems while saving a large number of images,
>     aurelien> which I guess was not so clear... I've worked a bit on
>     aurelien> it and I think the questions seem clearer to me now. I
>     aurelien> have paid particular attention to the threads in the
>     aurelien> user mailing list dealing with memory problems.
>
> What happens if you explicitly name the figure (ef figure(1))
>
> for i in range(indEnd):
>
>     figure(1)
>     subplot(221)
>     plot(ind, xx)
>
>     subplot(222)
>     X =3D rand(50,50)
>
>     imshow(X)
>     subplot(223)
>     scatter(rand(50), rand(50))
>     subplot(224)
>     pcolor(10*rand(50,50))
>
>     savefig('tmp%d' % i, dpi =3D 75)
>     close(1)
>
> See also the FAQ http://matplotlib.sourceforge.net/faq.html#LEAKS
> which shows the canonical way to make multiple plots to prevent leaks,
> namely, pairing a close with each figure creation?
>
> I'm not sure why you are seeing a problem right now, but my memory
> leak test script memleak_hawaii3.py does not appear to be leaking
>
> #!/usr/bin/env python
>
> import os, sys, time
> import matplotlib
> #matplotlib.interactive(True)
> #matplotlib.use('Cairo')
> matplotlib.use('Agg')
> from pylab import *
>
>
> def report_memory(i):
>     pid =3D os.getpid()
>     a2 =3D os.popen('ps -p %d -o rss,sz' % pid).readlines()
>     print i, '  ', a2[1],
>     return int(a2[1].split()[1])
>
>
>
> # take a memory snapshot on indStart and compare it with indEnd
>
> indStart, indEnd =3D 30, 201
> for i in range(indEnd):
>
>     figure(1); clf()
>
>     subplot(221)
>     t1 =3D arange(0.0, 2.0, 0.01)
>     y =3D sin(2*pi*t1)
>     plot(t1,y,'-')
>     plot(t1, rand(len(t1)), 's', hold=3DTrue)
>
>
>     subplot(222)
>     X =3D rand(50,50)
>
>     imshow(X)
>     subplot(223)
>     scatter(rand(50), rand(50), s=3D100*rand(50), c=3Drand(50))
>     subplot(224)
>     pcolor(10*rand(50,50))
>     savefig('tmp%d' % i, dpi =3D 75)
>     close(1)
>
>     val =3D report_memory(i)
>     if i=3D=3DindStart: start =3D val # wait a few cycles for memory us=
age to
> stabilize
>
> end =3D val
> print 'Average memory consumed per loop: %1.4fk bytes\n' %
> ((end-start)/float(indEnd-indStart))
>
>