From: Stephan D. <ste...@gm...> - 2007-03-16 10:53:27
|
I'm a longtime user of WebWare. I just stumbled over some pretty strange problem. The Application running on Webware is a kind of Document Management system that can contain quite huge files (ISO images). When somebody is downloading such a file and cancels the download, the application eats up more and more memory. It looks like the application tries to deliver the content, but, instead of delivering it to the client (who's not longer there), caches everything in memory. This is the newest version of WebWare. I've done the following change: In WebKit/ThreadedAppServer.py 741 def flush(self): 742 """Flush stream. 743 744 Calls `ASStreamOut.ASStreamOut.flush`, and if that returns True 745 (indicating the buffer is full enough) then we send data from 746 the buffer out on the socket. 747 748 """ 749 result = ASStreamOut.flush(self) 750 if result: # a True return value means we can send 751 reslen = len(self._buffer) 752 sent = 0 753 while sent < reslen: 754 try: 755 sent += self._socket.send( 756 self._buffer[sent:sent+8192]) 757 except socket.error, e: 758 if e[0] == errno.EPIPE: # broken pipe 759 pass 760 elif hasattr(errno, 'ECONNRESET') \ 761 and e[0] == errno.ECONNRESET: 762 pass 763 else: 764 print "StreamOut Error: ", e 765 break 766 self.pop(sent) I've changed line 759 from 'pass' to 'raise', which now gives ugly tracebacks in the logs, but other than that, does what should happen anyway: cancel the download. Stephan P.S.: I know this is quite a political issue, but is there a chance to change from tabs to spaces in Webware code? |
From: Christoph Z. <ci...@on...> - 2007-03-16 15:23:00
|
Stephan Diehl schrieb: > I'm a longtime user of WebWare. I just stumbled over some pretty > strange problem. The Application running on Webware is a kind of > Document Management system that can contain quite huge files (ISO > images). When somebody is downloading such a file and cancels the > download, the application eats up more and more memory. It looks like > the application tries to deliver the content, but, instead of > delivering it to the client (who's not longer there), caches > everything in memory. This is the newest version of WebWare. > > I've done the following change: > In WebKit/ThreadedAppServer.py > > 741 def flush(self): > 742 """Flush stream. > ... > 753 while sent < reslen: > 754 try: > 755 sent += self._socket.send( > 756 self._buffer[sent:sent+8192]) > 757 except socket.error, e: > 758 if e[0] == errno.EPIPE: # broken pipe > 759 pass > 760 elif hasattr(errno, 'ECONNRESET') \ > 761 and e[0] == errno.ECONNRESET: > 762 pass > 763 else: > 764 print "StreamOut Error: ", e > 765 break > 766 self.pop(sent) > > I've changed line 759 from 'pass' to 'raise', which now gives ugly > tracebacks in the logs, but other than that, does what should happen > anyway: cancel the download. Hi Stephan, thanks for the bug report. I think you're right, this is a problem. By the way, on Windows you get an ECONNABORTED error in the same situation, not an EPIPE error. So I think ECONNABORTED should be treated the same way as the other two errors in the flush method. For a fix of the problem, I suggest the following: In ASSStreamOut.py: Above the definition of the ASStreamOut class, insert: class ConnectionAbortedError(Exception): pass Below, replace the 2 lines beginning: assert not self._closed, ... with: if self._closed: raise ConnectionAbortedError In ThreadedAppServer.py: Above the definition of the ThreadedAppServer class, insert: from ASStreamOut import ConnectionAbortedError In the TASASStreamOut.flush() method, insert this line before the break statement: self._closed = True To avoid the ugly tracebacks, insert the following lines between the call of handler.handleRequest() and the following except:, indented with the correct amount (both "excepts" on the same level): except ConnectionAbortedError: if self._verbose: sys.stdout.write('%5i %s connection aborted\n' % (handler._requestID, timestamp()['pretty'])) If you confirm that this fixes your problems and nobody disagrees, I will fix it this way in the trunk. > P.S.: I know this is quite a political issue, but is there a chance to > change from tabs to spaces in Webware code? I think this question should go directly to Chuck, our BDFL who had been strongly against this in the past. I vote for a change as well. -- Christoph |
From: Stephan D. <ste...@gm...> - 2007-03-16 16:13:58
|
Hallo Christoph, thanks for that. I've cheated a bit, I'm still using Webware-0.9.1. When trying to upgrade my Application to Webware-0.9.2, I realized that the Application breaks. I'll need to investigate this first (I'm using a homegrown JSONRpc Handler). Alternativly, I could change my Webware-0.9.1 according to your suggestions. I'll definatelly get back to you about this. Stephan Christoph Zwerschke wrote: > Stephan Diehl schrieb: >> I'm a longtime user of WebWare. I just stumbled over some pretty >> strange problem. The Application running on Webware is a kind of >> Document Management system that can contain quite huge files (ISO >> images). When somebody is downloading such a file and cancels the >> download, the application eats up more and more memory. It looks like >> the application tries to deliver the content, but, instead of >> delivering it to the client (who's not longer there), caches >> everything in memory. This is the newest version of WebWare. > > > > I've done the following change: > > In WebKit/ThreadedAppServer.py > > > > 741 def flush(self): > > 742 """Flush stream. > > ... > > 753 while sent < reslen: > > 754 try: > > 755 sent += self._socket.send( > > 756 self._buffer[sent:sent+8192]) > > 757 except socket.error, e: > > 758 if e[0] == errno.EPIPE: # broken pipe > > 759 pass > > 760 elif hasattr(errno, 'ECONNRESET') \ > > 761 and e[0] == errno.ECONNRESET: > > 762 pass > > 763 else: > > 764 print "StreamOut Error: ", e > > 765 break > > 766 self.pop(sent) > > > > I've changed line 759 from 'pass' to 'raise', which now gives ugly > > tracebacks in the logs, but other than that, does what should happen > > anyway: cancel the download. > > Hi Stephan, > > thanks for the bug report. I think you're right, this is a problem. > > By the way, on Windows you get an ECONNABORTED error in the same > situation, not an EPIPE error. So I think ECONNABORTED should be treated > the same way as the other two errors in the flush method. > > For a fix of the problem, I suggest the following: > > In ASSStreamOut.py: > > Above the definition of the ASStreamOut class, insert: > > class ConnectionAbortedError(Exception): > pass > > Below, replace the 2 lines beginning: > > assert not self._closed, ... > > with: > > if self._closed: raise ConnectionAbortedError > > In ThreadedAppServer.py: > > Above the definition of the ThreadedAppServer class, insert: > >>from ASStreamOut import ConnectionAbortedError > > In the TASASStreamOut.flush() method, insert this line > before the break statement: > > self._closed = True > > To avoid the ugly tracebacks, insert the following lines between > the call of handler.handleRequest() and the following except:, > indented with the correct amount (both "excepts" on the same level): > > except ConnectionAbortedError: > if self._verbose: > sys.stdout.write('%5i %s connection aborted\n' > % (handler._requestID, timestamp()['pretty'])) > > If you confirm that this fixes your problems and nobody disagrees, I > will fix it this way in the trunk. > > > P.S.: I know this is quite a political issue, but is there a chance to > > change from tabs to spaces in Webware code? > > I think this question should go directly to Chuck, our BDFL who had been > strongly against this in the past. I vote for a change as well. > > -- Christoph > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Webware-discuss mailing list > Web...@li... > https://lists.sourceforge.net/lists/listinfo/webware-discuss > |
From: Christoph Z. <ci...@on...> - 2007-03-16 16:57:55
|
Stephan Diehl wrote: > I've cheated a bit, I'm still using Webware-0.9.1. When trying to > upgrade my Application to Webware-0.9.2, I realized that the > Application breaks. It would be great if you examine what breaks the application, since I want to publish a new release very soon. If there is a bug, I'd like to fix it first. (Some bugs in 0.9.2 have already been fixed, so it's time to get a bug-fix release 0.9.3 out as soon as possible.) -- Christoph |
From: Stephan D. <ste...@gm...> - 2007-03-16 19:18:18
|
Christoph Zwerschke wrote: > Stephan Diehl wrote: >> I've cheated a bit, I'm still using Webware-0.9.1. When trying to >> upgrade my Application to Webware-0.9.2, I realized that the >> Application breaks. > > It would be great if you examine what breaks the application, since I > want to publish a new release very soon. If there is a bug, I'd like to > fix it first. (Some bugs in 0.9.2 have already been fixed, so it's time > to get a bug-fix release 0.9.3 out as soon as possible.) Since my Application problem is most probably homegrown, I've followed up the original problem with the broken downloads. I'm doing more or less what's described in http://wiki.w4py.org/filestreamingandcontentdisposition.html In addition to your suggestions, I've also replaced the assert not self._closed, "Stream Already Closed" line in the write method (ASStreamOut.py) with if self._closed: raise ConnectionAbortedError catching the Error in ThreadedAppServer (as in your suggestion) does not supress the Traceback. It looks like the Traceback is generated in Application.py within the 'handleExceptionInTransaction'. For my special case it's probably best, to put my response.write inside some try/except. Stephan > > -- Christoph > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Webware-discuss mailing list > Web...@li... > https://lists.sourceforge.net/lists/listinfo/webware-discuss > |
From: Christoph Z. <ci...@on...> - 2007-03-17 00:25:03
|
Stephan Diehl wrote: > catching the Error in ThreadedAppServer (as in your suggestion) does not > supress the Traceback. It looks like the Traceback is generated in > Application.py within the 'handleExceptionInTransaction'. > For my special case it's probably best, to put my response.write inside > some try/except. You're right, that did not suffice. It worked only for pages < 8192 bytes, and the the transaction resources were not closed properly. I have now checked in a much cleaner and complete fix. -- Christoph |
From: Stephan D. <ste...@gm...> - 2007-03-19 09:10:13
|
Christoph Zwerschke wrote: > Stephan Diehl wrote: >> I've cheated a bit, I'm still using Webware-0.9.1. When trying to >> upgrade my Application to Webware-0.9.2, I realized that the >> Application breaks. > > It would be great if you examine what breaks the application, since I > want to publish a new release very soon. If there is a bug, I'd like to > fix it first. (Some bugs in 0.9.2 have already been fixed, so it's time > to get a bug-fix release 0.9.3 out as soon as possible.) Never been easier to fix a breakage: I've just tried it with the svn Webware trunk, and it works perfectly again. Stephan > > -- Christoph > > > ------------------------------------------------------------------------- > Take Surveys. Earn Cash. Influence the Future of IT > Join SourceForge.net's Techsay panel and you'll get the chance to share your > opinions on IT & business topics through brief surveys-and earn cash > http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV > _______________________________________________ > Webware-discuss mailing list > Web...@li... > https://lists.sourceforge.net/lists/listinfo/webware-discuss > |