Hello all,
I had a massive memory leak in some of my code. It would basically end up
using up all 1GB of my RAM or more if I don't kill the application. I
managed to finally figure out which portion of the code was causing the leak
(with great difficulty) and have a little example which exposes the leak. I
am using numpy-0.9.8 and I'm wondering if perhaps this is already fixed in
1.0b2. Run this through valgrind with appropriate options (I used the
recommended valgrind_py.sh that I found on scipy's site somewhere) and this
will leak 100kB. Increase the xrange on the big loop and you can watch the
memory increase over time in top.
The interesting thing is that the only difference between the leaky and
non-leaky code is:
if not adjacencyMatrix[anInt2,anInt1] == 0: (leaky)
vs.
if not adjacencyMatrix[anInt2][anInt1] == 0: (non-leaky)
however another way to make the leaky code non-leaky is to change
anArrayOfInts to just be [1]
Here's the code:
from numpy import array
def leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
ys = set()
for aList in aListOfArrays:
for anInt1 in anArrayOfInts:
for anInt2 in aList:
if not adjacencyMatrix[anInt2,anInt1] == 0:
ys.add(anInt1)
return ys
def nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix):
ys = set()
for aList in aListOfArrays:
for anInt1 in anArrayOfInts:
for anInt2 in aList:
if not adjacencyMatrix[anInt2][anInt1] == 0:
ys.add(anInt1)
return ys
if __name__ == "__main__":
for i in xrange(10000):
aListOfArrays = [[0, 1]]
anArrayOfInts = array([1])
adjacencyMatrix = array([[0,1],[1,0]])
#COMMENT OUT ONE OF THE 2 LINES BELOW
#bar = nonLeakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)
bar = leakyCode(aListOfArrays, anArrayOfInts, adjacencyMatrix)
--
David Grant
http://www.davidgrant.ca
|
