From: Geoff T. <gta...@na...> - 2000-11-29 23:08:16
|
I'm interested in modifying the CGI adapter (and mod_python adapter) so that if the WebKit app server is not currently available, they will retry for some period of time (say, 30 seconds) before giving up. My rationale is that sometimes you need to be able to restart the running app server to incorporate fixes, and it would be nice if all the users noticed was a delay, rather than an error message. I've already done this in my copy of the mod_python adapter, and I'm quite happy with the results. Is there any reason NOT to use retry in the adapters? If so, I could make it a configurable option. Otherwise, I'd rather just make retry the standard behavior. Comments? -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Chuck E. <ec...@mi...> - 2000-11-29 23:59:25
|
I think there's no reason why it shouldn't be standard, but at the same time it does need to be configurable. I think the configuration is needed because if the app server really isn't there, then you would expect two things at some point: * the user gets a reasonable error message (not a python exception like now) * the site admin gets an e-mail about the problem The two options that come to mind are: * retryDelay in seconds * maxRetries With values of perhaps, 20 and 6? BTW This also points out that ideally adapters should inherit the Adapter class so that we can consolidate common functionality. I think OneShotAdapter is currently the only one the does so. Note that Adapter looks for a config file in Configs/ with the same name as the class. Hence the existence of OneShotAdapter.config. -Chuck At 05:56 PM 11/29/2000 -0500, Geoff Talvola wrote: >I'm interested in modifying the CGI adapter (and mod_python adapter) so >that if the WebKit app server is not currently available, they will >retry for some period of time (say, 30 seconds) before giving up. My >rationale is that sometimes you need to be able to restart the running >app server to incorporate fixes, and it would be nice if all the users >noticed was a delay, rather than an error message. I've already done >this in my copy of the mod_python adapter, and I'm quite happy with the >results. > >Is there any reason NOT to use retry in the adapters? If so, I could >make it a configurable option. Otherwise, I'd rather just make retry >the standard behavior. Comments? > >-- > > >- Geoff Talvola > Parlance Corporation > gtalvola@NameConnector.com > >_______________________________________________ >Webware-discuss mailing list >Web...@li... >http://lists.sourceforge.net/mailman/listinfo/webware-discuss |
From: Tom S. <tsc...@li...> - 2000-11-30 12:47:25
|
Chuck Esterbrook wrote: > > I think there's no reason why it shouldn't be standard, but at the same > time it does need to be configurable. I think the configuration is needed > because if the app server really isn't there, then you would expect two > things at some point: > > * the user gets a reasonable error message (not a python exception like > now) > * the site admin gets an e-mail about the problem > > The two options that come to mind are: > * retryDelay in seconds > * maxRetries > > With values of perhaps, 20 and 6? What about starting the OneShot-Adapter of another safe installation? Maybe also a plugin-Architecture for modpHandler.py (in case of mod_python) could allow people to implement strategies for handling it the way they like it, e.g. first retry it (as above), when that fails, send a static page with OneShot or whatever you like.. In any case you should always get a reaction from the appserver framework, even when just restarting it. I restarted Zope to many time after product-updates and this is really annoying, because in the starting phase your site is down (and loading 500 MB Data.fs, as in my case, really takes some time!) If the webserver crashes then you are lost but also have a real problem. :-) -- Tom Schwaller http://www.linux-magazin.de http://www.linux-community.de |
From: Chuck E. <ec...@mi...> - 2000-12-01 04:01:47
|
At 02:44 PM 11/30/2000 +0100, Tom Schwaller wrote: >Chuck Esterbrook wrote: > > > > I think there's no reason why it shouldn't be standard, but at the same > > time it does need to be configurable. I think the configuration is needed > > because if the app server really isn't there, then you would expect two > > things at some point: > > > > * the user gets a reasonable error message (not a python exception like > > now) > > * the site admin gets an e-mail about the problem > > > > The two options that come to mind are: > > * retryDelay in seconds > > * maxRetries > > > > With values of perhaps, 20 and 6? > >What about starting the OneShot-Adapter of another >safe installation? > >Maybe also a plugin-Architecture for modpHandler.py >(in case of mod_python) could allow people to >implement strategies for handling it the way >they like it, e.g. first retry it (as above), when >that fails, send a static page with OneShot or whatever >you like.. That's a good point about using OneShot as a fail safe in case a "fancier" adapter fails. There might be issues with sessions, since other adapters typically prefer the memory session store while OneShot requires the file based one. In any case, I have noted this in the TO DO >In any case you should always get a reaction from >the appserver framework, even when just restarting it. >I restarted Zope to many time after product-updates >and this is really annoying, because in the starting >phase your site is down (and loading 500 MB Data.fs, >as in my case, really takes some time!) Sounds like a good reason not use a system that packs everything in one whopping 500MB binary image. Reminds me of Smalltalk, which had its good points, although that obviously wasn't one of them. -Chuck |
From: Tom S. <tom...@li...> - 2000-12-01 12:08:11
|
Chuck Esterbrook wrote: > >What about starting the OneShot-Adapter of another > >safe installation? > > > >Maybe also a plugin-Architecture for modpHandler.py > >(in case of mod_python) could allow people to > >implement strategies for handling it the way > >they like it, e.g. first retry it (as above), when > >that fails, send a static page with OneShot or whatever > >you like.. > > That's a good point about using OneShot as a fail safe in case a "fancier" > adapter fails. There might be issues with sessions, since other adapters > typically prefer the memory session store while OneShot requires the file > based one. jep, when I wrote the message I was thinking about this kind of problems too. One has to keep im mind, that it is a failsafe-solution, not the normal case. I guess I would change the color of the website and write somewhere: We are in a maintainance state and somethings might not be correct.. > In any case, I have noted this in the TO DO cool.. > >In any case you should always get a reaction from > >the appserver framework, even when just restarting it. > >I restarted Zope to many time after product-updates > >and this is really annoying, because in the starting > >phase your site is down (and loading 500 MB Data.fs, > >as in my case, really takes some time!) > > Sounds like a good reason not use a system that packs everything in one > whopping 500MB binary image. Reminds me of Smalltalk, which had its good > points, although that obviously wasn't one of them. Yes, indeed! Yesterday I put all the Zope-Article-data out of www.linux-magazin.de and serve it now with LocalFS. You can do that with most of the DTML-methods and not really use ZODB, but then you really donn't need Zope for most of it.. -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Chuck E. <ec...@mi...> - 2000-12-01 13:11:12
|
At 01:07 PM 12/1/2000 +0000, Tom Schwaller wrote: >One small WebWare showstoper right now is the missing >User Authentification and Security stuff and a Database >layer (connection poling and things like that. >MYSQL as a start is ok..). Is the MiddleKit supposed to >do that? One could of course implement something simple, >but in the long run this is a waste of time, because it >belongs in the WebKit itself. This 2 pieces and I'm ready >to migrate part of out websites.. MiddleKit maps object and relational data in a manner such that you primarily work with Python objects whose persistence in a SQL database is largely automated. It does not do anything with connection pooling at this point. There is such a class floating around in the message archives. Perhaps we need a component named DBKit or DBUtils that provides connection pooling and whatever other DB enhancements people need. Regarding security and authentication, I have been using Apache's .htaccess and htpasswd stuff. What kind of security and authentication enhancements do people want to see in Webware? e.g., what would these features be and how would they interact with WebKit? -Chuck |
From: Tom S. <tom...@li...> - 2000-12-03 08:44:08
Attachments:
DBPool.py
|
Chuck Esterbrook wrote: > > At 01:07 PM 12/1/2000 +0000, Tom Schwaller wrote: > >One small WebWare showstoper right now is the missing > >User Authentification and Security stuff and a Database > >layer (connection poling and things like that. > >MYSQL as a start is ok..). Is the MiddleKit supposed to > >do that? One could of course implement something simple, > >but in the long run this is a waste of time, because it > >belongs in the WebKit itself. This 2 pieces and I'm ready > >to migrate part of out websites.. > > MiddleKit maps object and relational data in a manner such that you > primarily work with Python objects whose persistence in a SQL database is > largely automated. It does not do anything with connection pooling at this > point. quite ambitious. Hope we have something to test soon :) Connections between tables are quit difficult to get right. I once saw the Open Source Tool POSSL (http://www.possl.org) in action, where you visually select the tables, make a selection of the fields per drag and dropand the rest is for you. It even detects the connnections between tables (if not, you can give hints). > There is such a class floating around in the message archives. I've found it, applied the 2 proposed patches and put it in Webware/WebKit/Cans/DBPool.py (DBPool.py file is attached). Then I teste it according to another posting (Examples/DB.py) from ExamplePage import ExamplePage import MySQLdb class DB(ExamplePage): def writeContent(self): self.writeln('<p>Welcome to the DBPooling example</p>') pool = self.getCan('DBPoolCan', 'DBPool', 'application', MySQLdb, 5, host='localhost', user='..', passwd='..', db='...') db = pool.getConnection() cursor = db.cursor() cursor.execute("SELECT realname FROM users WHERE userid < 25") for realname in cursor.fetchall(): self.writeln('%s<br>' % realname) db.close() Works fine (my first Can based SQL example :-)). But then I wanted it to know: /usr/sbin/ab -n 1000 -c 2 http://localhost/wpy/Examples/DB.py Requests per second: 48.28 WOW! Then I overloaded the site with /usr/sbin/ab -n 1000 -c 10 http://localhost/wpy/Examples/DB.py Requests per second: 31.40 WOW. Still very good compared to the same Zope example (I even did a loop (for in in range(15)...) and still got amazing resultas on my Dual Celeron, 256 MB RAM maschine). Unfortunately under this heavy load I also got the following exception: Traceback (innermost last): File "Application.py", line 283, in dispatchRequest self.handleGoodURL(transaction) File "Application.py", line 382, in handleGoodURL self.respond(transaction) File "Application.py", line 455, in respond transaction.respond() File "Transaction.py", line 93, in respond self._servlet.respond(self) File "HTTPServlet.py", line 38, in respond method(self, trans) File "Page.py", line 33, in respondToGet self._respond(transaction) File "Page.py", line 52, in _respond self.writeHTML() File "Page.py", line 100, in writeHTML self.writeBody() File "SidebarPage.py", line 56, in writeBody self.writeContent() File "Examples/DB.py", line 10, in writeContent db = pool.getConnection() File "Cans/DBPool.py", line 53, in _threadsafe_getConnection con = PooledConnection(self, self.connections[self.nextCon]) IndexError: list index out of range There seems to be a problem in the proposed implementation. I guess, it should allocate 5 connections and use them as before, but under heavy load it should allocate 5 more conections (and 5 more,...) when necessary. Anybody this kind of improvements at hand? Generally http://PoolMan.sourceforge.net/PoolMan/index.html is a very ood Java-implementation (Open Source) of a JDBC compliant connection pool (and even object pool ), which could serve as code basis. > Perhaps we need a component named DBKit or DBUtils that provides connection > pooling and whatever other DB enhancements people need. > > Regarding security and authentication, I have been using Apache's .htaccess > and htpasswd stuff. What kind of security and authentication enhancements > do people want to see in Webware? e.g., what would these features be and > how would they interact with WebKit? Probably I think to much in term of Zope Object publishing here, because you have accees to the methods and attributes there, but I prefer my classes to know what a user can do and what not and to instatiate the user data (e.g. from an SQL-DB) in memory when the user logs in. A general user/group permission framework (for people who want it) seems natural. The Zope code is quite instructive here, but not optional. WebWare could make that optional (some bases classes to use or not to use ), which is much better, because in many cases you do not need it in such a sophisticated way. ah, I forgot: the more I play with WebWare, the more I like it. Well done , guys :-) P.S. Did anybody try to get WebWare running with Jython? -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Chuck E. <ec...@mi...> - 2000-12-03 14:29:19
|
At 09:43 AM 12/3/2000 +0000, Tom Schwaller wrote: > > MiddleKit maps object and relational data in a manner such that you > > primarily work with Python objects whose persistence in a SQL database is > > largely automated. It does not do anything with connection pooling at this > > point. > >quite ambitious. Hope we have something to test soon :) "Quite ambitious"? You are correct, sir. :-) >Connections between tables are quit difficult to >get right. I once saw the Open Source Tool POSSL >(http://www.possl.org) in action, where you visually >select the tables, make a selection of the fields per >drag and dropand the rest is for you. It even detects >the connnections between tables (if not, you can give hints). I made an earlier design decision that has helped quite a bit, although it leaves MiddleKit (MK) with a significant caveat. MK lets you define a data-centric object model from which it can generate Python and SQL at design time, and perform operations with at run time. It does not allow you to take an existing legacy database and map that to objects like some other tools do. The advantage here is that MK is easier to implement and can have control over the implementation of the SQL schema. The disadvantage is that it can't be applied to legacy DBs. > > There is such a class floating around in the message archives. > >I've found it, applied the 2 proposed patches and put it in >Webware/WebKit/Cans/DBPool.py (DBPool.py file is attached). >Then I teste it according to another posting (Examples/DB.py) > >from ExamplePage import ExamplePage >import MySQLdb > >class DB(ExamplePage): > > def writeContent(self): > self.writeln('<p>Welcome to the DBPooling example</p>') > pool = self.getCan('DBPoolCan', 'DBPool', 'application', > MySQLdb, 5, host='localhost', user='..', > passwd='..', db='...') > db = pool.getConnection() > cursor = db.cursor() > cursor.execute("SELECT realname FROM users WHERE userid < 25") > for realname in cursor.fetchall(): > self.writeln('%s<br>' % realname) > db.close() > > >Works fine (my first Can based SQL example :-)). >But then I wanted it to know: > >/usr/sbin/ab -n 1000 -c 2 http://localhost/wpy/Examples/DB.py > >Requests per second: 48.28 >WOW! Then I overloaded the site with > >/usr/sbin/ab -n 1000 -c 10 http://localhost/wpy/Examples/DB.py >Requests per second: 31.40 Sweet. I guess this answers someone else's question on the list (er, maybe it was yours!). >WOW. Still very good compared to the same Zope example (I even >did a loop (for in in range(15)...) and still got amazing resultas >on my Dual Celeron, 256 MB RAM maschine). Unfortunately under >this heavy load I also got the following exception: > >Traceback (innermost last): [snip] > > Perhaps we need a component named DBKit or DBUtils that provides connection > > pooling and whatever other DB enhancements people need. > > > > Regarding security and authentication, I have been using Apache's .htaccess > > and htpasswd stuff. What kind of security and authentication enhancements > > do people want to see in Webware? e.g., what would these features be and > > how would they interact with WebKit? > >Probably I think to much in term of Zope Object publishing here, >because you have accees to the methods and attributes there, but >I prefer my classes to know what a user can do and what not and >to instatiate the user data (e.g. from an SQL-DB) in memory >when the user logs in. > >A general user/group permission framework (for people who want it) >seems natural. The Zope code is quite instructive here, but not >optional. WebWare could make that optional (some bases classes to use >or not to use ), which is much better, because in many cases you >do not need it in such a sophisticated way. All future enhancements not directly necessary for or intimately tied with serving servlets will reside outside of WebKit as plug-ins (e.g., Python packages that WebKit loads). Plug-ins can be replaced and disabled as users see fit. We have this architecture now. >ah, I forgot: the more I play with WebWare, the more >I like it. Well done , guys :-) Glad to hear it. >P.S. Did anybody try to get WebWare running with Jython? I haven't heard of anyone trying this yet. -Chuck |
From: Geoff T. <gta...@na...> - 2000-12-04 14:40:12
|
Tom Schwaller wrote: > WOW. Still very good compared to the same Zope example (I even > did a loop (for in in range(15)...) and still got amazing resultas > on my Dual Celeron, 256 MB RAM maschine). Unfortunately under > this heavy load I also got the following exception: > > Traceback (innermost last): > File "Application.py", line 283, in dispatchRequest > self.handleGoodURL(transaction) > File "Application.py", line 382, in handleGoodURL > self.respond(transaction) > File "Application.py", line 455, in respond > transaction.respond() > File "Transaction.py", line 93, in respond > self._servlet.respond(self) > File "HTTPServlet.py", line 38, in respond > method(self, trans) > File "Page.py", line 33, in respondToGet > self._respond(transaction) > File "Page.py", line 52, in _respond > self.writeHTML() > File "Page.py", line 100, in writeHTML > self.writeBody() > File "SidebarPage.py", line 56, in writeBody > self.writeContent() > File "Examples/DB.py", line 10, in writeContent > db = pool.getConnection() > File "Cans/DBPool.py", line 53, in _threadsafe_getConnection > con = PooledConnection(self, self.connections[self.nextCon]) > IndexError: list index out of range > > There seems to be a problem in the proposed implementation. > I guess, it should allocate 5 connections and use them > as before, but under heavy load it should allocate 5 more > conections (and 5 more,...) when necessary. Anybody this kind > of improvements at hand? Generally Looks like _threadsafe_getConnection is not itself thread-safe. Ironically, the "unthreadsafe" version looks thread-safe :-) One way to fix it is to use a Lock object. Add self.lock = threading.Lock() to the "elif dbModule.threadsafety >= 2:" part of __init__, then change _threadsafe_getConnection() to (untested): def _threadsafe_getConnection(self): self.lock.acquire() try: con = PooledConnection(self, self.connections[self.nextCon]) self.nextCon = self.nextCon + 1 if self.nextCon >= len(self.connections): self.nextCon = 0 return con finally: self.lock.release() -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Chuck E. <ec...@mi...> - 2000-12-04 18:07:49
|
At 06:18 PM 12/4/2000 +0100, Tom Schwaller wrote: >Is this code ready for CVS or will there be other under >the hood MySQL-mechanisms in MiddleKit? While I'm obviously a big fan of MiddleKit (MK), since I created it :-), I don't think it should eclipse other database techniques such as connection pools. e.g., if you just want a connection pool and you're not ready to embrace everything else that goes with MK, you should be able to do so. Would either you, Tom, or Geoff care to post the very latest complete version of the pool and example? My plan is to put this in a DBUtils component. -Chuck |
From: Chuck E. <ec...@mi...> - 2000-12-05 02:58:47
|
At 02:03 AM 12/5/2000 +0000, Tom Schwaller wrote: >DBPool.py and the dummy example DB.py is attached (I put DBPool.py >into the Cans directory because I needed the thread pool cross servlet >wise. Is there another way of reaching the same goal?) Once it's easier to subclass Application, then that will be an additional technique. For Python packages that don't require WebKit, there is also sometimes a need for a singular object, in which case I end up making a variable in a module for that purpose. For example, MiddleKit's ObjectStore module has a Store variable pointing to the global object store (although design-wise you can have multiple stores in the same process if you so choose). -Chuck |
From: Chuck E. <ec...@mi...> - 2000-12-05 12:19:56
|
At 01:09 PM 12/5/2000 +0000, Tom Schwaller wrote: >So should there be a connection pool variable in DBPool.py, so one can >use > >from MiscUtils.DBPool import pool > >or something similar? You could do that. For example, if you have a common super class for your pages called SitePage, it could do something like: class SitePage(Page): def __init__(self): Page.__init__(self) self._dbPool = None def dbPool(self): ''' Returns a single, global instance of a database connection pool (as defined in MiscUtils.DBPool). ''' if self._dbPool is None: from MiscUtils import DBPool if not hasattr(DBPool, 'Pool'): DBPool.Pool = DBPool.DBPool(args) self._dbPool = DBPool.Pool return self._dbPool Now all your pages can say "self.dbPool()". Or you could copy DBPool.py to WebKit/Cans. Or you could implement Can registration before I do. -Chuck |
From: Geoff T. <gta...@na...> - 2000-12-05 15:54:51
|
Chuck Esterbrook wrote: > You could do that. For example, if you have a common super class for your > pages called SitePage, it could do something like: > > class SitePage(Page): > > def __init__(self): > Page.__init__(self) > self._dbPool = None > > def dbPool(self): > ''' Returns a single, global instance of a database connection > pool (as defined in MiscUtils.DBPool). ''' > if self._dbPool is None: > from MiscUtils import DBPool > if not hasattr(DBPool, 'Pool'): > DBPool.Pool = DBPool.DBPool(args) > self._dbPool = DBPool.Pool > return self._dbPool > > Now all your pages can say "self.dbPool()". > Or you could copy DBPool.py to WebKit/Cans. > Or you could implement Can registration before I do. > > -Chuck I would wrap the DBPool initialization code with a lock so that two threads don't try to create the DBPool at the same time. How about: _lock = threading.Lock() ... def dbPool(self): ''' Returns a single, global instance of a database connection pool (as defined in MiscUtils.DBPool). ''' if self._dbPool is None: _lock.acquire() try: if self._dbPool is None: from MiscUtils import DBPool if not hasattr(DBPool, 'Pool'): DBPool.Pool = DBPool.DBPool(args) self._dbPool = DBPool.Pool finally: _lock.release() return self._dbPool -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Chuck E. <ec...@mi...> - 2000-12-05 16:25:30
|
Excellent point. At 10:58 AM 12/5/2000 -0500, Geoff Talvola wrote: >Chuck Esterbrook wrote: > > > You could do that. For example, if you have a common super class for your > > pages called SitePage, it could do something like: > > > > class SitePage(Page): > > > > def __init__(self): > > Page.__init__(self) > > self._dbPool = None > > > > def dbPool(self): > > ''' Returns a single, global instance of a database connection > > pool (as defined in MiscUtils.DBPool). ''' > > if self._dbPool is None: > > from MiscUtils import DBPool > > if not hasattr(DBPool, 'Pool'): > > DBPool.Pool = DBPool.DBPool(args) > > self._dbPool = DBPool.Pool > > return self._dbPool > > > > Now all your pages can say "self.dbPool()". > > Or you could copy DBPool.py to WebKit/Cans. > > Or you could implement Can registration before I do. > > > > -Chuck > >I would wrap the DBPool initialization code with a lock so that two threads >don't try to create the DBPool at the same time. How about: > > >_lock = threading.Lock() > >... > > def dbPool(self): > ''' Returns a single, global instance of a database connection pool >(as defined in MiscUtils.DBPool). ''' > if self._dbPool is None: > _lock.acquire() > try: > if self._dbPool is None: > from MiscUtils import DBPool > if not hasattr(DBPool, 'Pool'): > DBPool.Pool = DBPool.DBPool(args) > self._dbPool = DBPool.Pool > finally: > _lock.release() > return self._dbPool > >-- > > >- Geoff Talvola > Parlance Corporation > gtalvola@NameConnector.com > >_______________________________________________ >Webware-discuss mailing list >Web...@li... >http://lists.sourceforge.net/mailman/listinfo/webware-discuss |
From: Tom S. <tom...@li...> - 2000-12-01 12:08:15
|
hi all, One small WebWare showstoper right now is the missing User Authentification and Security stuff and a Database layer (connection poling and things like that. MYSQL as a start is ok..). Is the MiddleKit supposed to do that? One could of course implement something simple, but in the long run this is a waste of time, because it belongs in the WebKit itself. This 2 pieces and I'm ready to migrate part of out websites.. Nevertheless I will present WebKit in my next column (60. Tom's README) about application servers :-). WebWare looks very well designed and has a big potential. It's time will come :-) We implemented a small signal slot module which I plan to use for updating other parts of the website (eg. in Memory), when something has changed (example: authenticated user posts comment and gets Community Points for that. At least 10 other parts of the website need to make updates now and get the signal (with data), thas something has changed. With that you have a quite sophisticated cache mechanism which is upgradable. You just connect new parts of the website to old signals. It's running, but has a memory leak because of weak referneces. Right now we are incorporating http://PyWeakReference.sourceforge.net/ or http://www.lemburg.com/files/python/mxProxy.html I can post them here if people thinks this is appropriate. There was also a dicussion about it on comp.lang.python a few days ago. "Python Servlets" talking together is very important to me, because my community-site is very dynamic (every peace changes constantly) I also have to look at Cans (CanContainer.py CanFactory.py), unfortunately there is no real useful web-example right no. -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Geoff T. <gta...@na...> - 2000-12-01 15:32:33
|
You're right, although I worry that adding the config file stuff would slow down the CGI adapter noticeably. Anyhow, when I have some time I'll work on this, unless someone else gets to it first. Chuck Esterbrook wrote: > I think there's no reason why it shouldn't be standard, but at the same > time it does need to be configurable. I think the configuration is needed > because if the app server really isn't there, then you would expect two > things at some point: > > * the user gets a reasonable error message (not a python exception like > now) > * the site admin gets an e-mail about the problem > > The two options that come to mind are: > * retryDelay in seconds > * maxRetries > > With values of perhaps, 20 and 6? > > BTW This also points out that ideally adapters should inherit the Adapter > class so that we can consolidate common functionality. I think > OneShotAdapter is currently the only one the does so. > > Note that Adapter looks for a config file in Configs/ with the same name as > the class. Hence the existence of OneShotAdapter.config. > > -Chuck -- - Geoff Talvola Parlance Corporation gtalvola@NameConnector.com |
From: Chuck E. <ec...@mi...> - 2000-12-01 16:17:26
|
At 10:35 AM 12/1/2000 -0500, Geoff Talvola wrote: >You're right, although I worry that adding the config file stuff would slow >down the CGI adapter noticeably. WebKit.cgi already has the overhead of launching the Python interpreter and reading several modules in every time. And there's the actual round trip network access, checking the timestamp on the servlet, running the servlet... I don't think adding the config file in the context everything else will slow it down by more than a percent. And for those needing real speed, they have 4 options (AOLserver, mod_python, mod_snake, FastCGI). >Anyhow, when I have some time I'll work on this, unless someone else gets >to it first. Cool. I'm out this weekend, but on Monday I hope to go through all of Vladimir's patches. -Chuck |
From: Tom S. <tom...@li...> - 2000-12-03 22:15:17
|
Chuck Esterbrook wrote: > > At 09:43 AM 12/3/2000 +0000, Tom Schwaller wrote: > > > MiddleKit maps object and relational data in a manner such that you > > > primarily work with Python objects whose persistence in a SQL database is > > > largely automated. It does not do anything with connection pooling at this > > > point. > > > >quite ambitious. Hope we have something to test soon :) > > "Quite ambitious"? You are correct, sir. :-) :-)) > >Connections between tables are quit difficult to > >get right. I once saw the Open Source Tool POSSL > >(http://www.possl.org) in action, where you visually > >select the tables, make a selection of the fields per > >drag and dropand the rest is for you. It even detects > >the connnections between tables (if not, you can give hints). > > I made an earlier design decision that has helped quite a bit, although it > leaves MiddleKit (MK) with a significant caveat. MK lets you define a > data-centric object model from which it can generate Python and SQL at > design time, and perform operations with at run time. It does not allow you > to take an existing legacy database and map that to objects like some other > tools do. > > The advantage here is that MK is easier to implement and can have control > over the implementation of the SQL schema. The disadvantage is that it > can't be applied to legacy DBs. thanks for the info.. > > > There is such a class floating around in the message archives. > > > >I've found it, applied the 2 proposed patches and put it in > >Webware/WebKit/Cans/DBPool.py (DBPool.py file is attached). > >Then I teste it according to another posting (Examples/DB.py) > > > >from ExamplePage import ExamplePage > >import MySQLdb > > > >class DB(ExamplePage): > > > > def writeContent(self): > > self.writeln('<p>Welcome to the DBPooling example</p>') > > pool = self.getCan('DBPoolCan', 'DBPool', 'application', > > MySQLdb, 5, host='localhost', user='..', > > passwd='..', db='...') > > db = pool.getConnection() > > cursor = db.cursor() > > cursor.execute("SELECT realname FROM users WHERE userid < 25") > > for realname in cursor.fetchall(): > > self.writeln('%s<br>' % realname) > > db.close() > > > > > >Works fine (my first Can based SQL example :-)). > >But then I wanted it to know: > > > >/usr/sbin/ab -n 1000 -c 2 http://localhost/wpy/Examples/DB.py > > > >Requests per second: 48.28 > >WOW! Then I overloaded the site with > > > >/usr/sbin/ab -n 1000 -c 10 http://localhost/wpy/Examples/DB.py > >Requests per second: 31.40 > > Sweet. I guess this answers someone else's question on the list (er, maybe > it was yours!). jep, but the exeption it throws is still mysterious.. > All future enhancements not directly necessary for or intimately tied with > serving servlets will reside outside of WebKit as plug-ins (e.g., Python > packages that WebKit loads). Plug-ins can be replaced and disabled as users > see fit. We have this architecture now. Nevertheless I hope, that somebody with more connection pooling know-how than myself, writes such a beast, because high performance database examples will probably be the first thing people new to WebKit ask for. I'll port some of my website stuff in the coming weeks to see if I can get what I want, but see no real problems there. Is the above Can example correct? If you do not have cookies enabled then URL rewriting should be a fallback solution like in the Java case. There was an example on this list some thime ago, how to do that (i I remember it correctly), but hey, everybody is welcome to contribute small code snippets on this list (that's the way I learn :-)) > >ah, I forgot: the more I play with WebWare, the more > >I like it. Well done , guys :-) > > Glad to hear it. jep. more people should know about it, but this takes some time.. > >P.S. Did anybody try to get WebWare running with Jython? > > I haven't heard of anyone trying this yet. it could work, maybe I'll try that myself in a free minute... I could access EJBs in that way and also use RMI, which would be pretty cool. Where do you think one could contribute code right now? Some wishes? -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Chuck E. <ec...@mi...> - 2000-12-04 03:43:11
|
At 11:14 PM 12/3/2000 +0000, Tom Schwaller wrote: >Is the above Can example correct? Not sure. Haven't used cans much. Maybe Jay Love, the Can author will speak up. >If you do not have cookies enabled then URL rewriting >should be a fallback solution like in the Java case. >There was an example on this list some thime ago, >how to do that (i I remember it correctly), but hey, >everybody is welcome to contribute small code snippets >on this list (that's the way I learn :-)) I don't recall seeing that solution posted. For now, it's cookies or the highway. >Where do you think one could contribute code right now? >Some wishes? There is sometimes a Future.html document with the various components. There is a beefy one for WebKit. Take a look there. -Chuck |
From: Tom S. <tsc...@li...> - 2000-12-04 16:21:59
|
Geoff Talvola wrote: > > Tom Schwaller wrote: > > > WOW. Still very good compared to the same Zope example (I even > > did a loop (for in in range(15)...) and still got amazing resultas > > on my Dual Celeron, 256 MB RAM maschine). Unfortunately under > > this heavy load I also got the following exception: > > > > Traceback (innermost last): > > File "Application.py", line 283, in dispatchRequest > > self.handleGoodURL(transaction) > > File "Application.py", line 382, in handleGoodURL > > self.respond(transaction) > > File "Application.py", line 455, in respond > > transaction.respond() > > File "Transaction.py", line 93, in respond > > self._servlet.respond(self) > > File "HTTPServlet.py", line 38, in respond > > method(self, trans) > > File "Page.py", line 33, in respondToGet > > self._respond(transaction) > > File "Page.py", line 52, in _respond > > self.writeHTML() > > File "Page.py", line 100, in writeHTML > > self.writeBody() > > File "SidebarPage.py", line 56, in writeBody > > self.writeContent() > > File "Examples/DB.py", line 10, in writeContent > > db = pool.getConnection() > > File "Cans/DBPool.py", line 53, in _threadsafe_getConnection > > con = PooledConnection(self, self.connections[self.nextCon]) > > IndexError: list index out of range > > > > There seems to be a problem in the proposed implementation. > > I guess, it should allocate 5 connections and use them > > as before, but under heavy load it should allocate 5 more > > conections (and 5 more,...) when necessary. Anybody this kind > > of improvements at hand? Generally > > Looks like _threadsafe_getConnection is not itself thread-safe. Ironically, the "unthreadsafe" version looks > thread-safe :-) > > One way to fix it is to use a Lock object. Add > > self.lock = threading.Lock() > > to the "elif dbModule.threadsafety >= 2:" part of __init__, then change _threadsafe_getConnection() to > (untested): > > def _threadsafe_getConnection(self): > self.lock.acquire() > try: > con = PooledConnection(self, self.connections[self.nextCon]) > self.nextCon = self.nextCon + 1 > if self.nextCon >= len(self.connections): > self.nextCon = 0 > return con > finally: > self.lock.release() > We just had a discussion here about that problem and came to the same solution, so it's very recomforting to see your code here :-) thanks a lot. Seems to work nicely... Performancewise the following Code is as fast as the connection pooling example, but threads can produce inpredictably many connections if necessary.. from ExamplePage import ExamplePage import MySQLdb class DB1(ExamplePage): def __init__(self): ExamplePage.__init__(self) print 'connection1' self._db=MySQLdb.connect(user='...', passwd='...', db='...', unix_socket='/var/lib/mysql/mysql.sock') def writeContent(self): self.writeln('<P>Welcome to the DBPooling example!</P>') cursor = self._db.cursor() # can put that in __ init__ too cursor.execute("SELECT realname FROM users WHERE userid < 25") for realname in cursor.fetchall(): self.writeln('%s<br>' % realname) def __del__(self): self._db.close() Even getting a connection in writeContent has the same performance, but this is a MySQL feature. Other databases need much more time for the connection.. Is this code ready for CVS or will there be other under the hood MySQL-mechanisms in MiddleKit? -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de |
From: Tom S. <tom...@li...> - 2000-12-05 01:03:25
|
Chuck Esterbrook wrote: > > At 06:18 PM 12/4/2000 +0100, Tom Schwaller wrote: > >Is this code ready for CVS or will there be other under > >the hood MySQL-mechanisms in MiddleKit? > > While I'm obviously a big fan of MiddleKit (MK), since I created it :-), I > don't think it should eclipse other database techniques such as connection > pools. e.g., if you just want a connection pool and you're not ready to > embrace everything else that goes with MK, you should be able to do so. > > Would either you, Tom, or Geoff care to post the very latest complete > version of the pool and example? My plan is to put this in a DBUtils component. > > -Chuck DBPool.py and the dummy example DB.py is attached (I put DBPool.py into the Cans directory because I needed the thread pool cross servlet wise. Is there another way of reaching the same goal?) For my personal use I do something like: import MySQLdb class SQL: '''SQL base class''' def __init__(self, dbModule=MySQLdb, user='real data...', passwd='real data...', db='real data...'): self._db = dbModule.connect(user=user, passwd=passwd, db=db) self._cursor = self._db.cursor() def execute(self, command): self._cursor.execute(command) def fetchall(self): return self._cursor.fetchall() def __del__(self): self._db.close() ------------------------------------------------- from ExamplePage import ExamplePage from SQL import SQL import MySQLdb class DB2(ExamplePage, SQL): def __init__(self): ExamplePage.__init__(self) SQL.__init__(self) def writeContent(self): self.writeln('<P>Welcome to the DBPooling example!</P>') self.execute("SELECT realname FROM users WHERE userid < 25") for realname in self.fetchall(): self.writeln('%s<br>' % realname) # def __del__(self): # SQL.__del__(self) ---- In the longer run some database configuration option would be useful. e.g. DBDefault ='MySQLdb' hope this helps. Tomorrow I'll begin with a first "real world example" :-) -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Tom S. <tom...@li...> - 2000-12-05 12:11:07
|
Chuck Esterbrook wrote: > > At 02:03 AM 12/5/2000 +0000, Tom Schwaller wrote: > >DBPool.py and the dummy example DB.py is attached (I put DBPool.py > >into the Cans directory because I needed the thread pool cross servlet > >wise. Is there another way of reaching the same goal?) > > Once it's easier to subclass Application, then that will be an additional > technique. interesting, hope lo learn more about it by example. :-) > For Python packages that don't require WebKit, there is also sometimes a > need for a singular object, in which case I end up making a variable in a > module for that purpose. For example, MiddleKit's ObjectStore module has a > Store variable pointing to the global object store (although design-wise > you can have multiple stores in the same process if you so choose). So should there be a connection pool variable in DBPool.py, so one can use from MiscUtils.DBPool import pool or something similar? -- Tom Schwaller http://www.linux-community.de http://www.linux-magazin.de/ |
From: Chris A. <ca...@ca...> - 2000-12-05 18:41:55
|
Why not have the DBPool as a static member of your base class? Something like: class SitePage(Page): pool = None def __init__(self,**args): Page.__init__(self) if SitePage.pool == None: SitePage.pool = DBPool.DBPool(args) That way there is only one pool of connections for the whole system (assuming that everything inherits from SitePage). Otherwise each subclass gets its own pool, and you can quickly run out of connections. Cheers, Chris ----- Original Message ----- From: "Geoff Talvola" <gta...@na...> To: "Chuck Esterbrook" <ec...@mi...> Cc: <web...@li...> Sent: Tuesday, December 05, 2000 10:58 AM Subject: Re: [Webware-discuss] User Authentification and Security? > Chuck Esterbrook wrote: > > > You could do that. For example, if you have a common super class for your > > pages called SitePage, it could do something like: > > > > class SitePage(Page): > > > > def __init__(self): > > Page.__init__(self) > > self._dbPool = None > > > > def dbPool(self): > > ''' Returns a single, global instance of a database connection > > pool (as defined in MiscUtils.DBPool). ''' > > if self._dbPool is None: > > from MiscUtils import DBPool > > if not hasattr(DBPool, 'Pool'): > > DBPool.Pool = DBPool.DBPool(args) > > self._dbPool = DBPool.Pool > > return self._dbPool > > > > Now all your pages can say "self.dbPool()". > > Or you could copy DBPool.py to WebKit/Cans. > > Or you could implement Can registration before I do. > > > > -Chuck > > I would wrap the DBPool initialization code with a lock so that two threads > don't try to create the DBPool at the same time. How about: > > > _lock = threading.Lock() > > ... > > def dbPool(self): > ''' Returns a single, global instance of a database connection pool > (as defined in MiscUtils.DBPool). ''' > if self._dbPool is None: > _lock.acquire() > try: > if self._dbPool is None: > from MiscUtils import DBPool > if not hasattr(DBPool, 'Pool'): > DBPool.Pool = DBPool.DBPool(args) > self._dbPool = DBPool.Pool > finally: > _lock.release() > return self._dbPool > > -- > > > - Geoff Talvola > Parlance Corporation > gtalvola@NameConnector.com > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/mailman/listinfo/webware-discuss |
From: Chuck E. <ec...@mi...> - 2000-12-06 05:23:48
|
If you look more closely at the code I posted, there will only be one pool because it's taken from DBPool.Pool. The self._dbPool in my example is merely a cached reference to the global pool. I suppose I could have skipped that and just had the SitePage.dbPool() method return DBPool.Pool (after making sure it existed). In either case, the result is the same: one instance of DBPool. -Chuck At 01:42 PM 12/5/2000 -0500, Chris AtLee wrote: >Why not have the DBPool as a static member of your base class? Something >like: >class SitePage(Page): > pool = None > def __init__(self,**args): > Page.__init__(self) > if SitePage.pool == None: > SitePage.pool = DBPool.DBPool(args) > >That way there is only one pool of connections for the whole system >(assuming that everything inherits from SitePage). Otherwise each subclass >gets its own pool, and you can quickly run out of connections. > >Cheers, >Chris >----- Original Message ----- >From: "Geoff Talvola" <gta...@na...> >To: "Chuck Esterbrook" <ec...@mi...> >Cc: <web...@li...> >Sent: Tuesday, December 05, 2000 10:58 AM >Subject: Re: [Webware-discuss] User Authentification and Security? > > > > Chuck Esterbrook wrote: > > > > > You could do that. For example, if you have a common super class for >your > > > pages called SitePage, it could do something like: > > > > > > class SitePage(Page): > > > > > > def __init__(self): > > > Page.__init__(self) > > > self._dbPool = None > > > > > > def dbPool(self): > > > ''' Returns a single, global instance of a database connection > > > pool (as defined in MiscUtils.DBPool). ''' > > > if self._dbPool is None: > > > from MiscUtils import DBPool > > > if not hasattr(DBPool, 'Pool'): > > > DBPool.Pool = DBPool.DBPool(args) > > > self._dbPool = DBPool.Pool > > > return self._dbPool > > > > > > Now all your pages can say "self.dbPool()". > > > Or you could copy DBPool.py to WebKit/Cans. > > > Or you could implement Can registration before I do. > > > > > > -Chuck > > > > I would wrap the DBPool initialization code with a lock so that two >threads > > don't try to create the DBPool at the same time. How about: > > > > > > _lock = threading.Lock() > > > > ... > > > > def dbPool(self): > > ''' Returns a single, global instance of a database connection >pool > > (as defined in MiscUtils.DBPool). ''' > > if self._dbPool is None: > > _lock.acquire() > > try: > > if self._dbPool is None: > > from MiscUtils import DBPool > > if not hasattr(DBPool, 'Pool'): > > DBPool.Pool = DBPool.DBPool(args) > > self._dbPool = DBPool.Pool > > finally: > > _lock.release() > > return self._dbPool > > > > -- > > > > > > - Geoff Talvola > > Parlance Corporation > > gtalvola@NameConnector.com > > > > _______________________________________________ > > Webware-discuss mailing list > > Web...@li... > > http://lists.sourceforge.net/mailman/listinfo/webware-discuss > >_______________________________________________ >Webware-discuss mailing list >Web...@li... >http://lists.sourceforge.net/mailman/listinfo/webware-discuss |
From: Tripp L. <tl...@pe...> - 2000-12-06 05:53:02
|
I have some questions and observations relating to this thread, but they have more to do with general Python efficiency than with DBPool (thus the title change :) ). Yes, some of these things are pedantic. I'm not necessarily suggesting using them in practice, I'm just trying to get my head around some details :) I notice people keep the "if xxx.pool == None:" test in regardless of whether the pool attribute is an instance or a class attribute. If it's truly a singleton, is it more efficient to just initialize it at module load time than to keep checking its existence when you init instances? class SitePage (Page): pass SitePage.pool = DBPool.DBPool I notice that you're passing args in, which might be an argument against this "static initializer", but that's a little "strange" in the case presented below, since you might instantiate an object with one set of args, expecting its DBPool member to reflect those args, even though it's actually sharing a singleton that was originally constructed with another set of args... Second: are class / static attributes more costly to retrieve than instance attributes? This one is one that's particularly dear to me right now, since I'm trying to wrap my head around some sick foolishness for my own project (Selfish - <http:sourceforge.net/projects/selfish/> ). Anyway, when "pool" is an attribute of SitePage (the class, not the instance), this means that the interpreter first looks in the instance, then in the class (then inheritance, etc.), right? I guess that stuff's all done in fastish C, anyway, so it's not really that critical. Finally, a real, live, Webware question :) Does WW instantiate a new SitePage instance for each hit? The answer to that, of course, will inform how trivial or important my first question (about "if SitePage.pool...") is :) On Tue, 5 Dec 2000, Chuck Esterbrook wrote: > If you look more closely at the code I posted, there will only be one pool > because it's taken from DBPool.Pool. The self._dbPool in my example is > merely a cached reference to the global pool. > > I suppose I could have skipped that and just had the SitePage.dbPool() > method return DBPool.Pool (after making sure it existed). > > In either case, the result is the same: one instance of DBPool. > > -Chuck > > > At 01:42 PM 12/5/2000 -0500, Chris AtLee wrote: > >Why not have the DBPool as a static member of your base class? Something > >like: > >class SitePage(Page): > > pool = None > > def __init__(self,**args): > > Page.__init__(self) > > if SitePage.pool == None: > > SitePage.pool = DBPool.DBPool(args) > > > >That way there is only one pool of connections for the whole system > >(assuming that everything inherits from SitePage). Otherwise each subclass > >gets its own pool, and you can quickly run out of connections. > > > >Cheers, > >Chris > >----- Original Message ----- > >From: "Geoff Talvola" <gta...@na...> > >To: "Chuck Esterbrook" <ec...@mi...> > >Cc: <web...@li...> > >Sent: Tuesday, December 05, 2000 10:58 AM > >Subject: Re: [Webware-discuss] User Authentification and Security? > > > > > > > Chuck Esterbrook wrote: > > > > > > > You could do that. For example, if you have a common super class for > >your > > > > pages called SitePage, it could do something like: > > > > > > > > class SitePage(Page): > > > > > > > > def __init__(self): > > > > Page.__init__(self) > > > > self._dbPool = None > > > > > > > > def dbPool(self): > > > > ''' Returns a single, global instance of a database connection > > > > pool (as defined in MiscUtils.DBPool). ''' > > > > if self._dbPool is None: > > > > from MiscUtils import DBPool > > > > if not hasattr(DBPool, 'Pool'): > > > > DBPool.Pool = DBPool.DBPool(args) > > > > self._dbPool = DBPool.Pool > > > > return self._dbPool > > > > > > > > Now all your pages can say "self.dbPool()". > > > > Or you could copy DBPool.py to WebKit/Cans. > > > > Or you could implement Can registration before I do. > > > > > > > > -Chuck > > > > > > I would wrap the DBPool initialization code with a lock so that two > >threads > > > don't try to create the DBPool at the same time. How about: > > > > > > > > > _lock = threading.Lock() > > > > > > ... > > > > > > def dbPool(self): > > > ''' Returns a single, global instance of a database connection > >pool > > > (as defined in MiscUtils.DBPool). ''' > > > if self._dbPool is None: > > > _lock.acquire() > > > try: > > > if self._dbPool is None: > > > from MiscUtils import DBPool > > > if not hasattr(DBPool, 'Pool'): > > > DBPool.Pool = DBPool.DBPool(args) > > > self._dbPool = DBPool.Pool > > > finally: > > > _lock.release() > > > return self._dbPool > > > > > > -- > > > > > > > > > - Geoff Talvola > > > Parlance Corporation > > > gtalvola@NameConnector.com > > > > > > _______________________________________________ > > > Webware-discuss mailing list > > > Web...@li... > > > http://lists.sourceforge.net/mailman/listinfo/webware-discuss > > > >_______________________________________________ > >Webware-discuss mailing list > >Web...@li... > >http://lists.sourceforge.net/mailman/listinfo/webware-discuss > > _______________________________________________ > Webware-discuss mailing list > Web...@li... > http://lists.sourceforge.net/mailman/listinfo/webware-discuss > -- Joy-Loving * Tripp Lilley * http://stargate.eheart.sg505.net/~tlilley/ ------------------------------------------------------------------------------ "There were other lonely singers / in a world turned deaf and blind Who were crucified for what they tried to show. Their voices have been scattered by the swirling winds of time, 'Cause the truth remains that no one wants to know." - Kris Kristofferson, "To Beat the Devil" |