[Pypgsql-users] Memory leak?

[Sean R. <jaf...@tu...>]

Greetings.  I've been switching some of my code over to PyPgSQL and it
looks like I'm running into a memory leak in PyPgSQL.  I have two
different applications running on 6 different machines, all of them but
one exhibit growth in memory utilization.

I wish I had a solution, but at the moment I only have some data points:

   The two applications are not very similar.

   All systems are running Postgres 7.3 and 7.3.1 and PyPgSQL 2.3.  All
   systems are running Python 2.2.2, the 8.0 machines via Red Hat RPMs,
   the 7.3 machines via the python.org RPMs.

   4 systems are running Red Hat 7.3, 2 are running Red Hat 8.0.  The
   one machine that doesn't seem to be exhibiting the problem is an 8.0
   machine.  The 8.0 machines are both running the Red Hat "7.3-6" beta
   RPMs for postgresql.  The others are running 7.3.1 beta RPMs.

   I have converted one of the applications to use psycopg and PoPy
   (Ugh), and neither of these exhibit any real memory "leaking" on the
   the 8.0 machine that leaks with PyPgSQL.

   I haven't been able to use PyPgSQL 2.2 with PostgreSQL 7.3.

Code that exhibits the problem:

   from pyPgSQL import PgSQL as db
   import random

   co = db.connect('::pypgsqltest')
   cu = co.cursor()

   while 1:
      s = str(random.randrange(0, 1000000))
      cu.execute('SELECT id FROM a WHERE i = %s AND s = %s;', 3, s)

With a schema of:

   CREATE TABLE a (
      id serial,
      i integer,
      s text UNIQUE
      );

This code has been running for 3 CPU minutes and increased in RAM
utilization from 5.5MB to 12.6MB and continues to grow...

   [root@toomuchcoffee tmp]# ps -awlx | grep test3
   000     0 20463 17568  25   0 13208 10768 -     R    pts/1      3:03 python ./test3
   [root@toomuchcoffee tmp]# ps -awlx | grep test3
   000     0 20463 17568  25   0 14732 12300 -     R    pts/1      4:00 python ./test3
                                 ^^^^^ ^^^^^
                                 Total  RSS

Long-running jobs just tend to start swapping a lot after running for 8
to 12 hours full-speed, or weeks at slower rates.

For example, a minimal conversion of this program to use pyscopg
produces:

   000     0 20761 17457  16   0  5184 2740 schedu S    pts/0      0:02 python ./test3.psycopg
   000     0 20804 17457  15   0  5184 2740 schedu S    pts/0      1:06 python ./test3.psycopg

I wish I had an idea of where exactly the leek is happening, but I hope
this helps at least some...

Thanks,
Sean
-- 
 The question of whether a computer can think is no more interesting than the
 question of whether a submarine can swim."  -- Edsgar W. Dijkstra (1930-2002)
Sean Reifschneider, Inimitably Superfluous <ja...@tu...>
tummy.com, ltd. - Linux Consulting since 1995.  Qmail, Python, SysAdmin