Re: [Pypgsql-users] Memory leak?

[Billy G. A. <bil...@mu...>]

Sean Reifschneider wrote:
> Greetings.  I've been switching some of my code over to PyPgSQL and it
> looks like I'm running into a memory leak in PyPgSQL.  I have two
> different applications running on 6 different machines, all of them but
> one exhibit growth in memory utilization.
> 
> I wish I had a solution, but at the moment I only have some data points:
> 
>    The two applications are not very similar.
> 
>    All systems are running Postgres 7.3 and 7.3.1 and PyPgSQL 2.3.  All
>    systems are running Python 2.2.2, the 8.0 machines via Red Hat RPMs,
>    the 7.3 machines via the python.org RPMs.
> 
>    4 systems are running Red Hat 7.3, 2 are running Red Hat 8.0.  The
>    one machine that doesn't seem to be exhibiting the problem is an 8.0
>    machine.  The 8.0 machines are both running the Red Hat "7.3-6" beta
>    RPMs for postgresql.  The others are running 7.3.1 beta RPMs.
> 
>    I have converted one of the applications to use psycopg and PoPy
>    (Ugh), and neither of these exhibit any real memory "leaking" on the
>    the 8.0 machine that leaks with PyPgSQL.
> 
>    I haven't been able to use PyPgSQL 2.2 with PostgreSQL 7.3.

pyPgSQL 2.2 won't run with PostgreSQL 7.3.  You need at least pyPgSQL 2.3 to 
work with PostgreSQL 7.3

> Code that exhibits the problem:
> 
>    from pyPgSQL import PgSQL as db
>    import random
> 
>    co = db.connect('::pypgsqltest')
>    cu = co.cursor()
> 
>    while 1:
>       s = str(random.randrange(0, 1000000))
>       cu.execute('SELECT id FROM a WHERE i = %s AND s = %s;', 3, s)
> 
> With a schema of:
> 
>    CREATE TABLE a (
>       id serial,
>       i integer,
>       s text UNIQUE
>       );
> 
> This code has been running for 3 CPU minutes and increased in RAM
> utilization from 5.5MB to 12.6MB and continues to grow...
>     [...]

I created the table above and ran the given python code for 60 CPU minutes.
I did not experience any memory growth.  My memory started out at 5.3MB and 
stayed there.  Note: the table was empty for this test.

I then populated the table with ~1000 records and re-ran the test with the 
same result: no memory growth.

Can you tell my how many rows your sample table contains?

Does the sample code cause memory growth on all your machines?

I will inspect the code to see if anything obvious stands out.  Given the 
simplicity of the sample code, it would seem that the leak may be related to 
the libpq result object.

>     [...]
> I wish I had an idea of where exactly the leek is happening, but I hope
> this helps at least some...
> 
> Thanks,
> Sean

-- 
____       | Billy G. Allie    | Domain....: Bil...@mu...
|  /|      | 7436 Hartwell     | MSN.......: B_G...@em...
|-/-|----- | Dearborn, MI 48126|
|/  |LLIE  | (313) 582-1540    |