From: <Chu...@ya...> - 2003-11-06 09:21:50
|
There is a long standing problem that MiddleKit doesn't like MySQLdb 0.9.2,= although it gets along great with 0.9.1 and many versions prior. Under= MySQLdb 0.9.2, some of the MiddleKit tests yield "Too many connections": [snip] File "C:\All\echuck\Projects\Webware\MiddleKit\Run\MySQLObjectStore.py",= line 34, in newConnection conn =3D self.dbapiModule().connect(**args) File "C:\Python23\Lib\site-packages\MySQLdb\__init__.py", line 63, in= Connect return apply(Connection, args, kwargs) File "C:\Python23\Lib\site-packages\MySQLdb\connections.py", line 115, in= __init__ self._make_connection(args, kwargs2) File "C:\Python23\Lib\site-packages\MySQLdb\connections.py", line 41, in= _make_connection apply(super(ConnectionBase, self).__init__, args, kwargs) OperationalError: (1040, 'Too many connections') In fact, here are the results: RESULTS ------- succeeded MKBasic.mkmodel succeeded MKNone.mkmodel succeeded MKString.mkmodel succeeded MKDateTime.mkmodel succeeded MKDefaultMinMax.mkmodel succeeded MKTypeValueChecking.mkmodel succeeded MKInheritance.mkmodel succeeded MKInheritanceAbstract.mkmodel succeeded MKList.mkmodel succeeded MKObjRef.mkmodel succeeded MKObjRefReuse.mkmodel *** FAILED *** MKDelete.mkmodel *** FAILED *** MKDeleteMark.mkmodel succeeded MKMultipleStores.mkmodel *** FAILED *** MKMultipleThreads.mkmodel succeeded MKModelInh1.mkmodel succeeded MKModelInh2.mkmodel succeeded MKModelInh3.mkmodel This problem is so bad that one of the tests completely freezes and I have= to *kill* it with the Task Manager. (Each MK test is run in a separate= process, so the main Test.py keeps going with the other ones after any kind= of failure.) IIRC this problem happens under Linux as well, but it would be nice to= verify. Also, it happens with both Python 2.2.x and 2.3.x. I spoke with Andy Dustman, author of MySQLdb, awhile back but didn't= convince him it was a MySQLdb flaw (which is still my inclination, but not= proven) nor did the two of us come up with any good ideas. The impact of this problem goes like this: If a new user (or a user with a= new machine like me) downloads the latest stable versions of Python,= Webware and MySQLdb, then that user will have this problem. Yuk. Also, MySQLdb doesn't provide a 0.9.1 binary for Python 2.3 so the fix isn't= as easy as a download. Btw I tried this really cheesy fix: =09=09=09=09import gc, time =09=09=09=09reload(MySQLdb) =09=09=09=09gc.collect() =09=09=09=09time.sleep(0.5) with a loop of 5 attempts. No change in results. And my db server is not used by anything else. It's completely local,= private and freshly installed. I will follow up more on this, but for now it's getting late. I'd appreciate= any kind of help on this, even if it's as small as running the test suite= on platform X and reporting the results. Of course, solutions are welcome too. :-) So are speculations. And finally, instructions to run the test suite: * cd to MiddleKit/Tests * if you're in cvs: cvs upd -dP * If needed, edit TestRun.py line 23 from =09store =3D MySQLObjectStore() to =09store =3D MySQLObjectStore(user=3D'foo', passwd=3D'bar') * If needed, edit Test.py line 28 from =09=09return 'mysql' to =09=09return 'mysql -u foo -pbar' * Type: python Test.py MKDelete * or to run all tests, simply: python Test.py All tests take 75 seconds on my 3GHz P4. Sorry for the two edits. I plan to factor those out to a config file so you= can specify all the info in one place or via command line. -Chuck -- http://ChuckEsterbrook.com/ |
From: Tripp L. <tri...@pe...> - 2003-11-08 00:40:03
|
On Thu, 6 Nov 2003 Chu...@ya... wrote: > There is a long standing problem that MiddleKit doesn't like MySQLdb > 0.9.2, although it gets along great with 0.9.1 and many versions prior. > Under MySQLdb 0.9.2, some of the MiddleKit tests yield "Too many > connections": FYI, I've had this problem as well, under RH 8.0, Python 2.3 beta 1 (yes, I know), MySQL 3.23.46, and, of course, MySQLdb 0.9.2. My gut feeling is that this is a connection garbage collection issue *within* MySQLdb, and that it's not actually freeing up connections when it's supposed to (or not freeing up statement handles, or something like that). |
From: Tripp L. <tri...@pe...> - 2003-11-08 00:50:32
|
On Fri, 7 Nov 2003, Tripp Lilley wrote: > My gut feeling is that this is a connection garbage collection issue > *within* MySQLdb, and that it's not actually freeing up connections when > it's supposed to (or not freeing up statement handles, or something like > that). Looking at diffs between 0.9.1 and 0.9.2 just now, I'm even more inclined to go with this gut feeling, just given the quantity of "small" changes, as well as what appears to be a somewhat large-scale stylistic retooling of the MySQLdb internals. Naming conventions, internal data structures, etc., seem like they might have undergone some significant changes (but I'm just skimming, so I might be reading more into this than there is). In any case, it seems to me like writing a test case against MySQLdb that creates and then discards large numbers of connections would be a quick way to expose the behaviour for further study. Don't count on me to beat you to it this week, though :-) |
From: Chuck E. <Chu...@ya...> - 2003-11-08 01:28:12
|
On Fri, 7 Nov 2003 19:53:00 -0500 (EST), Tripp Lilley wrote: >=A0On Fri, 7 Nov 2003, Tripp Lilley wrote: > >>=A0My gut feeling is that this is a connection garbage collection >>=A0issue *within* MySQLdb, and that it's not actually freeing up >>=A0connections when it's supposed to (or not freeing up statement >>=A0handles, or something like that). >> >=A0Looking at diffs between 0.9.1 and 0.9.2 just now, I'm even more >=A0inclined to go with this gut feeling, just given the quantity of >=A0"small" changes, as well as what appears to be a somewhat large- >=A0scale stylistic retooling of the MySQLdb internals. Naming >=A0conventions, internal data structures, etc., seem like they might >=A0have undergone some significant changes (but I'm just skimming, so >=A0I might be reading more into this than there is). > >=A0In any case, it seems to me like writing a test case against >=A0MySQLdb that creates and then discards large numbers of connections >=A0would be a quick way to expose the behaviour for further study. >=A0Don't count on me to beat you to it this week, though :-) Agreed on all points. Thanks for the input. -Chuck -- http://ChuckEsterbrook.com/ |
From: Chuck E. <Chu...@ya...> - 2003-11-08 02:42:32
Attachments:
mysql-prob.py
|
Alright, here we go. The attached program exhibits the problem for me on= MySQLdb 0.9.2, but not 0.9.1. Admittedly, other variables changed too. Can a couple more people try it and report their results? A Linux build of= 0.9.3 beta would be especially interesting. Once I collect a few more data points, then I'll send this on to Andy= Dustman and see what he says... -Chuck -- http://ChuckEsterbrook.com/ |
From: Victor Ng <vn...@ma...> - 2003-11-09 09:33:18
|
FYI - I'm using MySQLdb 0.9.2 without WebWare and I get the out of connection error. Here' my exact stack trace: File "/usr/local/lib/python2.3/site-packages/MySQLdb/__init__.py", line 63, in Connect return apply(Connection, args, kwargs) File "/usr/local/lib/python2.3/site-packages/MySQLdb/connections.py", line 115, in __init__ self._make_connection(args, kwargs2) File "/usr/local/lib/python2.3/site-packages/MySQLdb/connections.py", line 41, in _make_connection apply(super(ConnectionBase, self).__init__, args, kwargs) OperationalError: (1040, 'Too many connections') vic On Saturday, November 8, 2003, at 01:42 PM, Chuck Esterbrook wrote: > Alright, here we go. The attached program exhibits the problem for me > on MySQLdb 0.9.2, but not 0.9.1. Admittedly, other variables changed > too. > > Can a couple more people try it and report their results? A Linux > build of 0.9.3 beta would be especially interesting. > > Once I collect a few more data points, then I'll send this on to Andy > Dustman and see what he says... > > -Chuck > -- > http://ChuckEsterbrook.com/ > <mysql-prob.py> |
From: Ben P. <be...@we...> - 2003-11-09 09:57:10
|
FWIW, if you close the connection before reassigning the variable to a new connection, your test script succeeds on my system (RedHat 7.3, MySQLdb 0.9.2, mysql 4.0.13-standard, python 2.2.3). Just add "conn.close()" at the end of the body of the for loop. It seems like the __del__ method isn't being called when the conn variable is reassigned to the new connection. It could be that MySQLdb keeps an internal reference to the connection so the reference count does drop to 0 when the conn variable is reassigned. I have not reviewed the MySQLdb code so that's only a guess. We have had a bunch of issues with memory leaks due to unclosed db connections using MySQLdb 0.9.2. It's now strict policy in code reviews to make sure all db connections and cursors are explicitly closed using <connection>.close(). - Ben > -----Original Message----- > From: web...@li... > [mailto:web...@li...]On Behalf Of Chuck > Esterbrook > Sent: Friday, November 07, 2003 6:43 PM > To: web...@li... > Subject: Re: [Webware-devel] MiddleKit & MySQLdb 0.9.2 problem > > > Alright, here we go. The attached program exhibits the problem > for me on MySQLdb 0.9.2, but not 0.9.1. Admittedly, other > variables changed too. > > Can a couple more people try it and report their results? A Linux > build of 0.9.3 beta would be especially interesting. > > Once I collect a few more data points, then I'll send this on to > Andy Dustman and see what he says... > > -Chuck > -- > http://ChuckEsterbrook.com/ > |
From: Chuck E. <Chu...@ya...> - 2003-11-09 21:10:56
|
On Sun, 9 Nov 2003 01:56:40 -0800, Ben Parker wrote: >=A0FWIW, if you close the connection before reassigning the variable >=A0to a new connection, your test script succeeds on my system (RedHat >=A07.3, MySQLdb 0.9.2, mysql 4.0.13-standard, python 2.2.3). Just add >=A0"conn.close()" at the end of the body of the for loop. > >=A0It seems like the __del__ method isn't being called when the conn >=A0variable is reassigned to the new connection. It could be that >=A0MySQLdb keeps an internal reference to the connection so the >=A0reference count does drop to 0 when the conn variable is ^ not >=A0reassigned. I have not reviewed the MySQLdb code so that's only a >=A0guess. > >=A0We have had a bunch of issues with memory leaks due to unclosed db >=A0connections using MySQLdb 0.9.2. It's now strict policy in code >=A0reviews to make sure all db connections and cursors are explicitly >=A0closed using <connection>.close(). In my real application, the method that creates the connection lives in a= library and returns the connection so that it can be optionally reused. So= I don't know if there is a straightforward fix in my situation. For now,= I'm using 0.9.1. -Chuck -- http://ChuckEsterbrook.com/ |
From: Ben P. <be...@we...> - 2003-11-10 06:29:39
|
> -----Original Message----- > From: web...@li... > [mailto:web...@li...]On Behalf Of Chuck > Esterbrook > Sent: Sunday, November 09, 2003 1:11 PM > To: web...@li... > Cc: an...@du... > Subject: RE: [Webware-devel] MiddleKit & MySQLdb 0.9.2 problem > > > On Sun, 9 Nov 2003 01:56:40 -0800, Ben Parker wrote: > > FWIW, if you close the connection before reassigning the variable > > to a new connection, your test script succeeds on my system (RedHat > > 7.3, MySQLdb 0.9.2, mysql 4.0.13-standard, python 2.2.3). Just add > > "conn.close()" at the end of the body of the for loop. > > > > It seems like the __del__ method isn't being called when the conn > > variable is reassigned to the new connection. It could be that > > MySQLdb keeps an internal reference to the connection so the > > reference count does drop to 0 when the conn variable is > ^ not > > reassigned. I have not reviewed the MySQLdb code so that's only a > > guess. > > > > We have had a bunch of issues with memory leaks due to unclosed db > > connections using MySQLdb 0.9.2. It's now strict policy in code > > reviews to make sure all db connections and cursors are explicitly > > closed using <connection>.close(). > > In my real application, the method that creates the connection > lives in a library and returns the connection so that it can be > optionally reused. So I don't know if there is a straightforward > fix in my situation. For now, I'm using 0.9.1. > > > -Chuck > -- > http://ChuckEsterbrook.com/ > Argh!, apologies for the dropped word there. funny how 3 letters can change the whole meaning of a sentence... Yeah, it's tough having optionally reusable connections. You may want to consider having a pool for reusable connections and a separate method for opening a single-use connection which will be closed by the thread that opens it. - Ben |