cx-oracle-users

RE: [cx-oracle-users] assistance in analyzing memory usage?

[Hancock, D. \(DHANCOCK\) <DHA...@ar...>]

Sorry to say I haven't got any solution to offer, but just a question: =
How does your memsnap() function work? I just hacked something like it =
to help us with other memory leak problems, but I'd like to see another =
implementation, because it's sure to be better than mine (mine saves =
sys.argv[0] to a string, and does a "ps auxww | grep 'that string' | =
grep -v grep"

Thanks, and
Cheers!
--
David Hancock | dha...@ar...


-----Original Message-----
From: cx-...@li... on behalf of Mark =
Harrison
Sent: Wed 9/28/2005 6:43 PM
To: cx-...@li...
Subject: [cx-oracle-users] assistance in analyzing memory usage?
=20
I'm seeing a memory leak when running this program which just
loops over a table and fetches its data.

Is there something I should be doing after a fetchmany to
free up some memory?

Attached below are some cleaned up output and the test program.
The "job" table has quite a few fields, including clob's.  Should
I be doing something special with them?

Many TIA!!
Mark

--
Mark Harrison
Pixar Animation Studios



USER       PID %CPU %MEM   VSZ  RSS
--- starting
mh       29770  0.0  0.0 54044 3260
--- import cx_Oracle
mh       29770  0.0  0.0 54044 3260
--- connect
mh       29770  0.0  0.1 75000 5640
--- cursor
mh       29770  0.0  0.2 141904 9860
--- arraysize
mh       29770  0.0  0.2 141904 9864
--- execute
mh       29770  0.0  0.2 141904 9864
--- fetchmany
mh       29770  0.0  0.6 195228 27308
mh       29770  0.0  2.0 218652 82924
mh       29770  0.0  2.5 236252 101356
mh       29770  0.0  2.5 236956 102428
mh       29770  0.0  2.5 237508 102968
mh       29770  0.0  2.5 238068 103696
mh       29770  0.0  2.5 238476 104088
mh       29770  0.0  2.5 238880 104540
mh       29770  0.0  2.5 239428 105108
mh       29770  0.0  2.6 239844 105628
mh       29770  0.0  2.6 240264 106048


import time
import os
os.system("ps auxww|head -n 1")
memsnap('starting')
memsnap('import cx_Oracle')
import cx_Oracle

user     =3D 'zzz'
passwd   =3D 'zzz'
connurl =3D 'racdb'

os.environ['TNS_ADMIN']=3D'/u0/mh/oracle'

memsnap('connect')
connection =3D cx_Oracle.connect(user, passwd, connurl)


memsnap('cursor')
cursor =3D connection.cursor() memsnap('arraysize')
cursor.arraysize =3D 10000

memsnap('execute')
cursor.execute('select * from job')
done =3D 0
tot =3D 0
print '---'
while not done:
     memsnap('fetchmany')
     result =3D cursor.fetchmany(10000)
     dir(result)
     tot +=3D len(result)
     if len(result) =3D=3D 0:
         done =3D 1
     result =3D []

cursor.close()

connection.close()


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, =
discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
cx-oracle-users mailing list
cx-...@li...
https://lists.sourceforge.net/lists/listinfo/cx-oracle-users

Re: [cx-oracle-users] assistance in analyzing memory usage?

[Mark H. <mh...@pi...>]

Hancock, David (DHANCOCK) wrote:
> Sorry to say I haven't got any solution to offer, but just a question: How does your memsnap() function work? I just hacked something like it to help us with other memory leak problems, but I'd like to see another implementation, because it's sure to be better than mine (mine saves sys.argv[0] to a string, and does a "ps auxww | grep 'that string' | grep -v grep"
> 
> Thanks, and
> Cheers!

Heh, yours is probably better than mine coz I did the same and just hard-coded the
executable name.

Here's a nice grep tip:  if you put the first character of the grep pattern
in square brackets you won't need the final grep -v, coz the string "[q]"
doesn't match the regular expression "[q]".

def memsnap(note):
     print '---', note
     os.system("ps auxww|grep '[q]2'")

If you don't mind a linux-only solution, you can open /proc/PID/status
where the memory stats are available in a nicely parsable form.

I've also been pointed to memusage on linux which will produce nice graphs.

HTH!
Mark

-- 
Mark Harrison
Pixar Animation Studio