From: Edmund L. <el...@in...> - 2002-10-26 17:45:41
|
On Sat, 2002-10-26 at 12:14, Edmund Lian wrote: > Is there a way to resolve this problem automatically, or will I have to > register a shutdownhandler to explicitly delete the global pool instance? Just as a follow up, I did try to register a clean up function with atexit, but this doesn't work. And, since the module isn't a part of any context, I don't see how I'd import and use Application.addShutdownHandler. On 10/26/2002 01:39:04 PM Jason Hildebrand wrote: >Hmmm... I suspect your problem would be fixed if the file descriptors >representing your database connections had the close-on-exec flag set >(man fcntl for details). > >It would be nice if the autoreload code could set all file descriptors >to close-on-exec before restarting, but I don't know if there's a way to >get a list of all open file descriptors. Anyone? Nuts, this is beyond my understanding of the situation... ...Edmund. |
From: Edmund L. <el...@in...> - 2002-10-26 22:34:32
|
On 10/26/2002 04:25:44 PM Ian Bicking wrote: >I suppose the problem is that you can't get to the Application object? Yes, this is the problem. Could the problem of leaking DB connections be at all related to how DBPool.py doesn't actually close connections on being deleted? I'm actually using multiple instances of a refactored version of DBPool.py to maintain multiple pools. The code doesn't actually close the connections when the instance is deleted. I recall some discussion of this about a year ago. At the time, it seemed OK to just let the database clean up connections after the pool instance is deleted. I didn't bother adding code to close the connections on a __del__ because it didn't seem important then. ...Edmund. |
From: Edmund L. <el...@in...> - 2002-10-28 13:42:51
|
On 10/26/2002 06:34:17 PM I wrote: >Could the problem of leaking DB connections be at all related to how DBPool.py >doesn't actually close connections on being deleted? I'm actually using >multiple instances of a refactored version of DBPool.py to maintain multiple >pools. The code doesn't actually close the connections when the instance is >deleted. OK, I've educated myself--sort of... I added code to close connections, and called the code from __del__ definitions. It doesn't help since the restart doesn't seem to call __del__ at all. A real shutdown of the app server does cause __del__ to be called. Why is this (just asking to understand what's going on)? ...Edmund. |
From: Stefan S. <ste...@tu...> - 2002-10-28 16:32:00
|
Hi Edmund On Mon, 28 Oct 2002, Edmund Lian wrote: > >Could the problem of leaking DB connections be at all related to how DBPool.py > >doesn't actually close connections on being deleted? I'm actually using > >multiple instances of a refactored version of DBPool.py to maintain multiple > >pools. The code doesn't actually close the connections when the instance is > >deleted. > > OK, I've educated myself--sort of... I added code to close connections, and > called the code from __del__ definitions. It doesn't help since the restart > doesn't seem to call __del__ at all. A real shutdown of the app server does > cause __del__ to be called. Why is this (just asking to understand what's > going on)? The actual __del__ will only be called when the object which contains the __del__ method is garbage-collected, i. e. when there are no more references to the object. It seems that the exec doesn't affect the existing references which is in line with the observations that started this discussion. Stefan |
From: Jason H. <ja...@pe...> - 2002-10-28 16:43:37
|
On Mon, 2002-10-28 at 10:31, Stefan Schwarzer wrote: > Hi Edmund > > On Mon, 28 Oct 2002, Edmund Lian wrote: > > >Could the problem of leaking DB connections be at all related to how DBPool.py > > >doesn't actually close connections on being deleted? I'm actually using > > >multiple instances of a refactored version of DBPool.py to maintain multiple > > >pools. The code doesn't actually close the connections when the instance is > > >deleted. > > > > OK, I've educated myself--sort of... I added code to close connections, and > > called the code from __del__ definitions. It doesn't help since the restart > > doesn't seem to call __del__ at all. A real shutdown of the app server does > > cause __del__ to be called. Why is this (just asking to understand what's > > going on)? > > The actual __del__ will only be called when the object which contains > the __del__ method is garbage-collected, i. e. when there are no more > references to the object. It seems that the exec doesn't affect the > existing references which is in line with the observations that > started this discussion. Correct: exec doesn't give the Python interpreter a chance to clean everything up; it re-loads the current process immediately (the call never returns). It would be better if we did give Python a chance to clean things up, though, so I'm going to change the implementation to not use exec(). I'll post more details later today. -- Jason D. Hildebrand ja...@pe... |
From: Ian B. <ia...@co...> - 2002-10-26 20:25:03
|
On Sat, 2002-10-26 at 12:45, Edmund Lian wrote: > > Is there a way to resolve this problem automatically, or will I have to > > register a shutdownhandler to explicitly delete the global pool instance? > > Just as a follow up, I did try to register a clean up function with atexit, > but this doesn't work. And, since the module isn't a part of any context, I > don't see how I'd import and use Application.addShutdownHandler. I suppose the problem is that you can't get to the Application object? That's annoying that it's not available -- while it's vaguely possible that you'd have two Application instances in the same process, I can't even think of a reason we'd want to keep that option open. So a reasonable fix for this problem would be to add some access to Application. I wonder if all open file objects are kept open, though? That might be a problem... I mean, not everything has to be perfect with auto-reload, since it's still only meant for development purposes. What we really want is for the Python interpreter to do a clean exit, calling all finalizers, and then restart. I don't know if you can do that with os.exec. Maybe we should ask comp.lang.python -- I'm sure we could get a definitive answer from someone. Ian |
From: Jason H. <ja...@pe...> - 2002-10-26 22:10:37
|
> I wonder if all open file objects are kept open, though? That might be > a problem... I mean, not everything has to be perfect with auto-reload, > since it's still only meant for development purposes. I checked on my system by looking in /proc/<PID>/fd for the appserver. I saw that I got a few extra open files after each restart, too (I think it was the database connections). MySQL didn't seem to mind, but I'd like to fix this if possible. I'm sure it would become a problem after a few hours of piled-up connections. > What we really want is for the Python interpreter to do a clean exit, > calling all finalizers, and then restart. I don't know if you can do > that with os.exec. Maybe we should ask comp.lang.python -- I'm sure we > could get a definitive answer from someone. I found this thread while googling, which shed a bit of light on the subject. http://mail.python.org/pipermail/python-list/2001-January/025343.html The python interpreter doesn't keep track of file descriptors; only the OS does. There's no portable way of getting a list of open files, which means that the only way to close all open files is to try to close every possible file descriptor, ignoring errors. Of course we want to leave 0,1, and 2 open (stdin,stdout,stderr); these _should_ be inherited by the new appserver. Unfortunately, when I stuck in code to do this (before the exec), the appserver process started spinning, and I had to kill -9 it. I've been trying to track down what's going wrong, and so far I've concluded that there are some pipes being opened that don't want to get closed (maybe it causes deadlock if you close them in the wrong order?). So far, I haven't been able to determine where they are being opened. Here's a typical listing of the appserver's open files: 0 -> /dev/null 1 -> /home/jdhildeb/projects/mobile/devapp/appserver.log 10 -> socket:[126562] 11 -> socket:[126564] 12 -> /usr/lib/python2.2/Cookie.py 2 -> /home/jdhildeb/projects/mobile/devapp/appserver.log 3 -> pipe:[9811] 4 -> /tmp/foo 5 -> /tmp/foo2 6 -> pipe:[126557] 7 -> pipe:[126557] 8 -> /tmp/foo5 9 -> /tmp/foo3 All of the "foo" entries are from my testing trying to find where the pipes are being opened, so you can ignore them. Numbers 6 and 7 cause the problems (i.e. you can close off all other descriptors without problems.). I'm going to pursue this further, since closing all file descriptors before the exec would be a fairly clean way of handling this. cheers, Jason -- Jason D. Hildebrand ja...@pe... |
From: Ian B. <ia...@co...> - 2002-10-26 22:36:36
|
Just a quick thought -- it might be easier to put this into the AppServer script, and say if Launch.py exits with a certain error code it will restart the AppServer (instead of just exiting). Then the process fully exits, and all files will be cleaned up by the OS. I'm pretty sure AppServer.bat could do this too (somehow... despite the lameness of batch programming). On Sat, 2002-10-26 at 17:10, Jason Hildebrand wrote: > > I wonder if all open file objects are kept open, though? That might be > > a problem... I mean, not everything has to be perfect with auto-reload, > > since it's still only meant for development purposes. > > I checked on my system by looking in /proc/<PID>/fd for the appserver. > I saw that I got a few extra open files after each restart, too (I think > it was the database connections). MySQL didn't seem to mind, but I'd > like to fix this if possible. I'm sure it would become a problem after > a few hours of piled-up connections. > > > What we really want is for the Python interpreter to do a clean exit, > > calling all finalizers, and then restart. I don't know if you can do > > that with os.exec. Maybe we should ask comp.lang.python -- I'm sure we > > could get a definitive answer from someone. > > I found this thread while googling, which shed a bit of light on the > subject. > > http://mail.python.org/pipermail/python-list/2001-January/025343.html > > The python interpreter doesn't keep track of file descriptors; only the > OS does. There's no portable way of getting a list of open files, which > means that the only way to close all open files is to try to close every > possible file descriptor, ignoring errors. Of course we want to leave > 0,1, and 2 open (stdin,stdout,stderr); these _should_ be inherited by > the new appserver. > > Unfortunately, when I stuck in code to do this (before the exec), the > appserver process started spinning, and I had to kill -9 it. I've been > trying to track down what's going wrong, and so far I've concluded that > there are some pipes being opened that don't want to get closed (maybe > it causes deadlock if you close them in the wrong order?). So far, I > haven't been able to determine where they are being opened. Here's a > typical listing of the appserver's open files: > > 0 -> /dev/null > 1 -> /home/jdhildeb/projects/mobile/devapp/appserver.log > 10 -> socket:[126562] > 11 -> socket:[126564] > 12 -> /usr/lib/python2.2/Cookie.py > 2 -> /home/jdhildeb/projects/mobile/devapp/appserver.log > 3 -> pipe:[9811] > 4 -> /tmp/foo > 5 -> /tmp/foo2 > 6 -> pipe:[126557] > 7 -> pipe:[126557] > 8 -> /tmp/foo5 > 9 -> /tmp/foo3 > > All of the "foo" entries are from my testing trying to find where the > pipes are being opened, so you can ignore them. > > Numbers 6 and 7 cause the problems (i.e. you can close off all other > descriptors without problems.). > > I'm going to pursue this further, since closing all file descriptors > before the exec would be a fairly clean way of handling this. > > cheers, > Jason > > -- > Jason D. Hildebrand > ja...@pe... |
From: Jason H. <ja...@pe...> - 2002-10-26 22:58:54
|
On Sat, 2002-10-26 at 17:37, Ian Bicking wrote: > Just a quick thought -- it might be easier to put this into the > AppServer script, and say if Launch.py exits with a certain error code > it will restart the AppServer (instead of just exiting). Then the > process fully exits, and all files will be cleaned up by the OS. I'm > pretty sure AppServer.bat could do this too (somehow... despite the > lameness of batch programming). This idea occurred to me, too. The only drawback to this is that the PID change would change all the time, but maybe that's not a problem. I'll persist a little more (with closing the file descriptors prior to exec()) and see if I can get it to work. If not, we can go the AppServer scripting route. -- Jason D. Hildebrand ja...@pe... |
From: Jason H. <ja...@pe...> - 2002-10-27 00:41:16
|
On Sat, 2002-10-26 at 17:58, Jason Hildebrand wrote: > This idea occurred to me, too. The only drawback to this is that the > PID change would change all the time, but maybe that's not a problem. > > I'll persist a little more (with closing the file descriptors prior to > exec()) and see if I can get it to work. If not, we can go the > AppServer scripting route. To follow up: I wasn't able to figure out where the mysterious pipes are coming from and why they cause the appserver to spin when I close them. So, I think the most sensible thing to do is use the exit code from the appserver to signal the AppServer script to start it again. I'll check in code to do this, but likely not until Monday. cheers, Jason -- Jason D. Hildebrand ja...@pe... |
From: Ian B. <ia...@co...> - 2002-10-27 07:14:03
|
On Sat, 2002-10-26 at 17:58, Jason Hildebrand wrote: > On Sat, 2002-10-26 at 17:37, Ian Bicking wrote: > > Just a quick thought -- it might be easier to put this into the > > AppServer script, and say if Launch.py exits with a certain error code > > it will restart the AppServer (instead of just exiting). Then the > > process fully exits, and all files will be cleaned up by the OS. I'm > > pretty sure AppServer.bat could do this too (somehow... despite the > > lameness of batch programming). > > This idea occurred to me, too. The only drawback to this is that the > PID change would change all the time, but maybe that's not a problem. But the PID of the AppServer script should remain, right? If you want to stop the AppServer, couldn't you still kill that PID (and then the child PID that is the actual AppServer will be killed along with it)...? Ian |
From: Jason H. <ja...@pe...> - 2002-10-27 15:03:26
|
On Sun, 2002-10-27 at 02:14, Ian Bicking wrote: > > This idea occurred to me, too. The only drawback to this is that the > > PID change would change all the time, but maybe that's not a problem. > > But the PID of the AppServer script should remain, right? If you want > to stop the AppServer, couldn't you still kill that PID (and then the > child PID that is the actual AppServer will be killed along with > it)...? Kills don't propogate to child-processes, so this wouldn't work. But the appserver will still write out it's PID to appserverpid.txt every time it is started. So stopping the server is simply kill `cat appserverpid.txt` just like it always has been. -- Jason D. Hildebrand ja...@pe... |