Thread: [SQLObject] RFP: SQLObject-Web ?
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: Stef T. <st...@um...> - 2009-02-01 21:17:51
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Everyone, Okay.. I have a question. I think (perhaps wrongly but) that most people are using SQLObject in a web fashion these days. To my limited sight, the code seems structured to exist in a more non-web fashion. What do I mean by this ? *) There is writeLocks - surely not required since most frameworks run the entire thread under a transaction anyway. Surely these are therefore, at best, depreciated in a web environment and, at worst, obsolete. It also makes pickle hard (obviously) cause I have to remove them from the object before storing into memcache and then assign a new one after a 'get' (of course, I have to do this with sqlmeta - -anyway-, but at least I can see the reason d'etre with that. very handy and cool) *) cache built in at a fundamental level - most web people would probably feel safer with memcache, currently I don't see anyway to disable this other than by hacking the get method inside SQLObject to not put items into it's cache (at least, that's the way I got memcache to be used and the inbuilt cache never used). Yes, I -can- subclass the object, but surely the base object should be 'light' and then a new SQLObject with caching derives from that. This would mean anyone could subclass from the base object and the FK/Join goodness without being forced to use SQLObjects caching. *) no query caching - seems strange to have object caching but no query caching, when query caching is generally where 'web' starts to notice major slowness. I have created a simple queryCache using a global inside the servlet/framework request to do this, and it seems to work pretty fine. *) connection pooling - I would have thought (again perhaps naively) that in a web environment, this should exist outside the ORM. In essence, something like pgpool-II or sequoia. There are whole applications designed to do this, and even at it's best, on a page load I notice that there are about 20 connections (did I mention that the pages in my website/environment can get pretty Object-Heavy ? :). So, is it insanity to propose that perhaps there is a sqlobject-web version ? missing writelocks (acid handled by the database), no caching built in (memcache), no connection pooling (again, other parts of the framework can handle this and argueably better/cleaner). This email should NOT be taken as flamebait or anger or anything other than a progressive/open email to start a discussion. If anyone takes any offense, then my deepest apologies, as none are meant in anyway. Regards Stef -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJhgrqANG7uQ+9D9URAppvAJwKZFTB9QVp0BnTPIlJphj2PUweLgCgizgO lt4o6qgrjgPBvmLICavJsk4= =xYqf -----END PGP SIGNATURE----- |
From: Oleg B. <ph...@ph...> - 2009-02-01 22:41:33
|
Hello! My company uses SQLObject solely for desktop applications. On Sun, Feb 01, 2009 at 03:49:46PM -0500, Stef Telford wrote: > *) There is writeLocks - surely not required since most frameworks > run the entire thread under a transaction anyway. I don't see how a web transaction is related to multithread-safe locks. Whatever a web transaction is, there are a lot of different web frameworks, and who guarantees it is thread-safe to touch shared variables? > *) cache built in at a fundamental level - most web people would > probably feel safer with memcache, currently I don't see anyway to > disable this Probably by writing a Cache class that doesn't cache. > *) no query caching What is a query in terms of an ORM? If 'document' is an SQLObject, attribute access like 'document.title' is implemented using a SQL query. How does query caching help? In any case - if query caching helps, SQLObject will benefit if somebody commits a code, right? > *) connection pooling - I would have thought (again perhaps > naively) that in a web environment, this should exist outside the ORM. It is easy to disable: connection._pool = None. > So, is it insanity to propose that perhaps there is a > sqlobject-web version? You are welcome to support one. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Stef T. <st...@um...> - 2009-02-01 23:25:13
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello Oleg, Oleg Broytmann wrote: > Hello! > > My company uses SQLObject solely for desktop applications. > Understandable, and this is pretty much how I was thinking SQLObject was designed/targetted for. This makes perfect sense, but in a web framework perhaps not as much. No offense meant ;) > On Sun, Feb 01, 2009 at 03:49:46PM -0500, Stef Telford wrote: >> *) There is writeLocks - surely not required since most >> frameworks run the entire thread under a transaction anyway. > > I don't see how a web transaction is related to multithread-safe > locks. Whatever a web transaction is, there are a lot of different > web frameworks, and who guarantees it is thread-safe to touch > shared variables? > well, from what I have seen over the years, most web frameworks that use an ORM do the following; start request handling in a thread create connection to db do processing commit connection/rollback capture end request I know that's -very- high level, but, there is never really any contention in such an environment and, therefore, there is no need for writeLocks. I am thinking of webware here, probably Turbogears from what I can recall and .. you get the idea. so, I am suggesting that the web edition get's -rid- of writeLocks :) >> *) cache built in at a fundamental level - most web people would >> probably feel safer with memcache, currently I don't see anyway >> to disable this > > Probably by writing a Cache class that doesn't cache. > yes.. I could do that, but then the 'web' edition would have all these references to 'cache' laying around that would be confusing for later people/programmers. I would prefer to keep code clean and simple if at all possible. >> *) no query caching > > What is a query in terms of an ORM? If 'document' is an SQLObject, > attribute access like 'document.title' is implemented using a SQL > query. How does query caching help? In any case - if query caching > helps, SQLObject will benefit if somebody commits a code, right? > True.. true. I have rudimentary queryCaching but, as I said, it only - -really- makes in the servlet/web response scenario above. I noticed that SQLObject issued a lot of similiar sql calls, again, mostly due to the object heavy pages we have/use. In -your- usage of SQLObject, the query cache would probably be a lot more pain than it is worth. We would have to hook into setattr on sqlojbect, get the class name and id, then walk over the queryCache checking if that update invalidates any of the cached queries.. major pain and probably not worth it. In -my- usage pattern, it makes a lot of sense. Sometimes a page calls .count() more than 20 times for the same query/object. Poor design ? perhaps, however I think that the 'path of least surprise' is that SQLObject would 'cache' that query and return the cached value since we are inside the same request/response cycle. Again, this is totally understandable given how -you- use SQLObject .. I am merely wandering if it may make sense to make a more .. well .. 'web-centric' version. I could be totally wrong here, of course, hence why it's a RFP ;) >> *) connection pooling - I would have thought (again perhaps >> naively) that in a web environment, this should exist outside the >> ORM. > > It is easy to disable: connection._pool = None. > interesting. Noted (and thank you) - I guess I am still stuck in request->response thinking. The connection seems to be screaming 'Singleton' or 'Borg' to me.. but.. perhaps that's jst me :) >> So, is it insanity to propose that perhaps there is a >> sqlobject-web version? > > You are welcome to support one. > Well, there's the rub.. I could fork a new version, and then spend most of my time playing catch-up with the 'official' releases of SQLObject .. but I don't think that really helps anyone. Is there any way of making a git branch and then before a release even flinging across an email/changeset ? I don't know your dev process's involved :D Thanks for the prompt reply btw :D Regards Stef > Oleg. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJhi9TANG7uQ+9D9URAhYJAJ42WZQe/gICkU+imbDvxFIiTjc4ywCfbLh1 n9+b02Wj0s1wJeTc2jX8ctA= =+tq3 -----END PGP SIGNATURE----- |
From: David T. <no...@op...> - 2009-02-01 23:57:46
|
On Sun, 2009-02-01 at 18:25 -0500, Stef Telford wrote: > Hello Oleg, > > Oleg Broytmann wrote: > > Hello! > > > > My company uses SQLObject solely for desktop applications. > > > Understandable, and this is pretty much how I was thinking SQLObject > was designed/targetted for. This makes perfect sense, but in a web > framework perhaps not as much. No offense meant ;) > > On Sun, Feb 01, 2009 at 03:49:46PM -0500, Stef Telford wrote: > >> *) There is writeLocks - surely not required since most > >> frameworks run the entire thread under a transaction anyway. > > > > I don't see how a web transaction is related to multithread-safe > > locks. Whatever a web transaction is, there are a lot of different > > web frameworks, and who guarantees it is thread-safe to touch > > shared variables? > > > well, from what I have seen over the years, most web frameworks that > use an ORM do the following; > start request handling in a thread > create connection to db > do processing > commit connection/rollback capture > end request > > I know that's -very- high level, but, there is never really any > contention in such an environment and, therefore, there is no need for > writeLocks. I am thinking of webware here, probably Turbogears from > what I can recall and .. you get the idea. so, I am suggesting that > the web edition get's -rid- of writeLocks :) This is also how some of my Pylons-based applications work, fwiw. I would definitely use this. > >> *) no query caching > > > > What is a query in terms of an ORM? If 'document' is an SQLObject, > > attribute access like 'document.title' is implemented using a SQL > > query. How does query caching help? In any case - if query caching > > helps, SQLObject will benefit if somebody commits a code, right? > > > True.. true. I have rudimentary queryCaching but, as I said, it only > - -really- makes in the servlet/web response scenario above. I noticed > that SQLObject issued a lot of similiar sql calls, again, mostly due > to the object heavy pages we have/use. In -your- usage of SQLObject, > the query cache would probably be a lot more pain than it is worth. We > would have to hook into setattr on sqlojbect, get the class name and > id, then walk over the queryCache checking if that update invalidates > any of the cached queries.. major pain and probably not worth it. > > In -my- usage pattern, it makes a lot of sense. Sometimes a page calls > .count() more than 20 times for the same query/object. Poor design ? > perhaps, however I think that the 'path of least surprise' is that > SQLObject would 'cache' that query and return the cached value since > we are inside the same request/response cycle. > > Again, this is totally understandable given how -you- use SQLObject .. > I am merely wandering if it may make sense to make a more .. well .. > 'web-centric' version. I could be totally wrong here, of course, hence > why it's a RFP ;) I'm less interested in this one, because I usually just use a per-request cache when I need one. |
From: Stef T. <st...@um...> - 2009-02-02 17:38:51
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oleg Broytmann wrote: > On Sun, Feb 01, 2009 at 06:25:07PM -0500, Stef Telford wrote: >> Oleg Broytmann wrote: >>> My company uses SQLObject solely for desktop applications. >>> >> Understandable, and this is pretty much how I was thinking >> SQLObject was designed/targetted for. This makes perfect sense, >> but in a web > > I don't think it is how and what it's designed for. Its original > author, Ian Bicking, AFAIK, had been working in a web environment. > >> framework perhaps not as much. No offense meant ;) > > No offence taken. :) > good show. I always get worried about people taking offense over e-mail. I find that tone is very hard to convey. >>>> *) There is writeLocks - surely not required since most >>>> frameworks run the entire thread under a transaction anyway. >>> I don't see how a web transaction is related to >>> multithread-safe locks. Whatever a web transaction is, there >>> are a lot of different web frameworks, and who guarantees it is >>> thread-safe to touch shared variables? >>> >> well, from what I have seen over the years, most web frameworks >> that use an ORM do the following; start request handling in a >> thread create connection to db do processing commit >> connection/rollback capture end request >> >> I know that's -very- high level, but, there is never really any >> contention in such an environment and, therefore, there is no >> need for writeLocks. I am thinking of webware here, probably >> Turbogears from what I can recall and .. you get the idea. so, I >> am suggesting that the web edition get's -rid- of writeLocks :) > > Still I don't understand. What do you mean by saying "there is no > contention"? If there are threads and there are shared variables - > the variables must be protected, right? Have I missed something? > There are global counters, e.g. in Alias - how would they fare if > different threads would want to increase them? > Well, at the start of every request, a connection is created to the database. Now, -any- rdbms worth it's salt basically creates an isolation level so that I can see -my- changes but no one else's changes, and it also 'locks' any items that are updated - thus avoiding update contention. The -database- should handle this in other words, rather than trying to put the problem onto the application to deal with this, let's let the database handle it. I assume that this is what the writeLocks are trying to do (stop different threads from updating the same item at the sametime). I understand that RDBMS rules out mySQL (not flamebait, merely saying that ACID isn't really "there" on mySQL to my liking). Perhaps I haven't dug far enough into sqlobject's code yet. but if all the objects are instantiated in one thread, the thread runs at the start of the request through until the end of the request before going back to the threading pool, then why is threading.lock() even needed ? how can another process 'jump' or assume thread control (assuming the framework doesn't call yield, which it doesn't ;) ? perhaps I misunderstand python's threading model. The globals table (as far as I can tell) contain a list of tables and expressions for sqlbuilder.. when would these change -without- me having to restart the webserver ? if I change the schema of an object in the database, I would expect to bounce the webservers, right ? >>>> *) cache built in at a fundamental level - most web people >>>> would probably feel safer with memcache, currently I don't >>>> see anyway to disable this >>> Probably by writing a Cache class that doesn't cache. >>> >> yes.. I could do that, but then the 'web' edition would have all >> these references to 'cache' laying around that would be confusing >> for later people/programmers. I would prefer to keep code clean >> and simple if at all possible. > > Another approach for SQLObject would be to grow an API to disable > caching, similar to how one can disable connection pooling. Setting > connection.cache to None, or a flag in sqlmeta. But the patch to > implement this would be rather big and have to be thoroughly > tested. > >>>> *) no query caching >>> What is a query in terms of an ORM? >> In -my- usage pattern, it makes a lot of sense. Sometimes a page >> calls .count() more than 20 times for the same query/object. Poor >> design ? perhaps, however I think that the 'path of least >> surprise' is that SQLObject would 'cache' that query and return >> the cached value since we are inside the same request/response >> cycle. > > Even inside one transaction the result of any .count() could > change. But if one is satisfied with caching results of such > queries... well, it is up to her/him, I don't mind having an API > for such caching. > The count should could change inside a transaction, but the web 'workflow' doesn't really mix page rendering and saving in one transaction. If a save requests come in, I would expect a new connection to be forked, all the work done in 'updating' in that one, and a 'read' connection which only EVER deals with 'reads'. I don't think this use case is really much different from how most web frameworks work. A mixed read+update style in a single connection, personally speaking, would seem like -making- work for myself ;) >>>> So, is it insanity to propose that perhaps there is a >>>> sqlobject-web version? >>> You are welcome to support one. >>> >> Well, there's the rub.. I could fork a new version, and then >> spend most of my time playing catch-up with the 'official' >> releases of SQLObject .. but I don't think that really helps >> anyone. Is there any way of making a git branch and then before a >> release even flinging across an email/changeset ? I don't know >> your dev process's involved :D > > Currently the developers are using Subversion. Worse then, the > development process is a bit stalled as I don't have spare time and > other developers are not active too. :( > ah, sorry to hear that. Hrm. alright. this does raise a certain amount of .. concern. I think this end up being the -real- problem (no offense meant again :). I don't mind tidying up and putting in memcache support and master/slave style connections for a web version, but I obviously don't want to play 'catch up' with every new release that you fellows do.. hrm. Regards Stef > Oleg. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJhy+gANG7uQ+9D9URAiAGAJ0atjrZ8hHjxQRW/oLES6ju57CewQCbB94l sREC4Wlep8lOhF5WlNCUAMk= =msNB -----END PGP SIGNATURE----- |
From: Stef T. <st...@um...> - 2009-02-02 20:07:50
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oleg Broytmann wrote: > On Mon, Feb 02, 2009 at 12:38:45PM -0500, Stef Telford wrote: >> Oleg Broytmann wrote: >>> On Sun, Feb 01, 2009 at 06:25:07PM -0500, Stef Telford wrote: >>> No offence taken. :) >> good show. I always get worried about people taking offense over >> e-mail. I find that tone is very hard to convey. > > I know. I did it many time in this mailing list - inadvertently > offended people. Different levels of English knowledge, different > vocabularies, different understanding... > Well, don't worry about your english, I am scottish - NO one understands me usually when I talk ;) I dare say, your english is better than (I am guessing) my russian (??) so, not a worry/concern. >>> Still I don't understand. What do you mean by saying "there is >>> no contention"? If there are threads and there are shared >>> variables - the variables must be protected, right? Have I >>> missed something? There are global counters, e.g. in Alias - >>> how would they fare if different threads would want to increase >>> them? >> Well, at the start of every request, a connection is created to >> the database. Now, -any- rdbms worth it's salt basically creates >> an isolation level so that I can see -my- changes but no one >> else's changes > > Aha, I see. We are talking about different concepts. You are > talking about database isolation. I am talking about python shared > variables. Those locks in the code protect python variables, not > database! Think about that very pool, connection._pool - nobody > (i.e., no other thread) should read it even less should write to it > until SQLObject finishes manipulating it. The same is true for > alias lock. > Ah, getting rid of the writelocks for connection pooling isn't a worry (I was thinking of getting rid of that and letting the framework deal with that instead). However, what is the alias lock for ? In the scenario I plan on using SQLObject in, would an alias lock even be required/needed ? alias's will stay 'fixed' at webserver/framework launch .. no ? >> I don't mind tidying up and putting in memcache support and >> master/slave style connections for a web version, but I obviously >> don't want to play 'catch up' with every new release that you >> fellows do.. hrm. > > But what was you plan from the beginning? If you were going to have > a forked version of SQLObject - wouldn't fork and catching up the > only option? You can add some code to the very SQLObject, but if > you want to remove some generic code - what is the way? > Actually, I -had- hoped that SQLObject could have writelocks enabled/disabled, same for connection pooling (which you tell me it does, which is great), caching enabled/disabled (the version inside SQLObject). The queryCaching I do currently by using python's 'global' in the servlet , I can't imagine that going into the main SQLObject (and it probably shouldn't). I could put that in a derived class from the framework Servlet itself. Let me jst say that I have had -bad- previous experience with forks of projects and libraries. Invariably they start to become more and more incompatible. I would prefer -not- to get into that situation if possible. Actually, is it possible to get (under subversion) a delta of patches from one version to another but cherry picking the changes to be made ? That's the one thing under git that would make things .. 'bearable' at least. Thank you for your time Oleg, I know my emails may seem to be strange, especially if english is not your first language. It is truly appreciated. Regards Stef > Oleg. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJh1KUANG7uQ+9D9URAqsjAJ9Y9z3O7b1hMt1ZwrpuX/mKsEmcwQCfVg89 05Vbk+mOwUeRTDqI+HxDnXQ= =1/yh -----END PGP SIGNATURE----- |
From: Stef T. <st...@um...> - 2009-02-03 02:39:09
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oleg Broytmann wrote: > On Mon, Feb 02, 2009 at 03:07:49PM -0500, Stef Telford wrote: >> alias's will stay 'fixed' at webserver/framework >> launch .. no ? > > Alias counter is increased on every instantiation of Alias(). > Stupid question here .. but .. why ? (sorry, I know that this could be a large answer but..) Also, whilst I am here, a couple more stupid questions, is there any flag in the SQLObject which tells me if the object was 'created' from the database or instantiated from the database/cache ? I ask only because it would be -awfully- handy to have .. I did look at _creating but, that's not the right item (obviously) Lastly (I promise) I am thinking of over-riding the _SO_finishCreate in the SQLObject subclass so that all database inserts go to the master rather than any slave. I would also have to override _SO_setValue and set (of course) .. any other methods in SQLObject that would have to be subclassed ? Regards Stef -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFJh65JANG7uQ+9D9URAjB2AJ9cCoeoKupnZnsFelu3Zt70w729WgCdE/0I rQgn6pwVF3N1p/DG+gAVa+w= =Qnqs -----END PGP SIGNATURE----- |
From: Stef T. <st...@um...> - 2009-02-03 19:45:01
|
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Oleg Broytmann wrote: > On Mon, Feb 02, 2009 at 09:39:06PM -0500, Stef Telford wrote: >> Oleg Broytmann wrote: >>> On Mon, Feb 02, 2009 at 03:07:49PM -0500, Stef Telford wrote: >>>> alias's will stay 'fixed' at webserver/framework launch .. no >>>> ? >>> Alias counter is increased on every instantiation of Alias(). >> Stupid question here .. but .. why ? (sorry, I know that this >> could be a large answer but..) > > Because SQLObject needs to generate unique alias names and a global > counter is the simplest way to do it. Think about joining a table > with itself - you need two different names for the table. > Thanks for your time once again Oleg. but. Sorry, perhaps I am being stupid here (always a possability) but, how many self joins are we expecting in a query ? I mean to say, isn't saying 'selfName = self.__class__.__name__ + "_%d" % int(random())' good enough ? I don't know about how mySQL planner would deal with a self join on a self join .. but I can guess (eg; NOT well) so .. surely we only need to worry about -1- self join per table in any query ? and since queries are always 'stand alone' we don't have to worry about leakage to other areas/queries, why not merely chose a random number and append ? Sorry, I guess I am not grasping the fine details here.. my apologies :) >> Also, whilst I am here, a couple more stupid questions, is there >> any flag in the SQLObject which tells me if the object was >> 'created' from the database or instantiated from the >> database/cache ? I ask only because it would be -awfully- handy >> to have .. I did look at _creating but, that's not the right item >> (obviously) > > There is no and I really fail to see a difference. Why do you want > to distinguish them? > Really ? Well, in my application, there is a lot of 'house keeping' that needs to be done after a new row is created in the database, and it seems crazy to call _setattr_ if the row has been grabbed from the cache (as any call to setattr would/should invalidate the object in the cache and force a reload from the database). Think of 'completeName' .. which is simply self.firstName + " " + self.lastName. I can think of a 'good reason' to only call -that- after the initial object has been created (I know that this should be a model function which jst returns the firstName + " " + lastName but, go with me here, I am trying to keep my example simple). If there is a 'newRowInDB' flag, then I can say something like; > def _init(self, id, connection=None, selectResults=None): ''' > Overload this method if you need to do anything special ''' > SQLObject._init(self, id, connection, selectResults) if > self.newRowInDB: self.completeName = self.firstName + " " + > self.lastName memcache.put("%s_%s" % (self.__class__.__name__, > self.id)) If we -dont- have that newRowInDB flag, then any cache hit will call the _setattr_ on completeName, which will thus (potentially) invalidate the object in the cache and cause another get for SQLObject .. which will cause _setattr_ to be called... round and round we go. I think even having a flag set on self somewhere inside the _SO_finishCreate would do the trick, but, I am assuming that this is the only place that creates new objects in the db. >> Lastly (I promise) I am thinking of over-riding the >> _SO_finishCreate in the SQLObject subclass so that all database >> inserts go to the master rather than any slave. I would also have >> to override _SO_setValue and set (of course) .. any other methods >> in SQLObject that would have to be subclassed ? > > No, these seem to be enough. > Danke. Regards Stef > Oleg. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD4DBQFJiJzsANG7uQ+9D9URAtgcAJjHdYSZ4UV4fHp9UwBFqQ4A8JJ1AJ9yKfAN gT7kfXEpCT/vivcHB6u+9A== =zN9m -----END PGP SIGNATURE----- |
From: Oleg B. <ph...@ph...> - 2009-02-03 20:45:43
|
On Tue, Feb 03, 2009 at 02:37:16PM -0500, Stef Telford wrote: > Oleg Broytmann wrote: > > Because SQLObject needs to generate unique alias names and a global > > counter is the simplest way to do it. Think about joining a table > > with itself - you need two different names for the table. > > Sorry, perhaps I am being stupid here (always a possability) but, how > many self joins are we expecting in a query ? I mean to say, isn't > saying 'selfName = self.__class__.__name__ + "_%d" % int(random())' > good enough ? Are you sure calling random() and int() is really so much faster than lock/increase counter/unlock? Are you sure you are optimizing a real stumbling block? > it seems crazy to call _setattr_ if the row has been grabbed from the > cache (as any call to setattr would/should invalidate the object in > the cache and force a reload from the database). What __setattr__ do you mean? SQLObject doesn't have __setattr__, so it doesn't invalidate object, and certainly doesn't reload it from DB. > Think of 'completeName' .. which is simply self.firstName + " " + > self.lastName. I can think of a 'good reason' to only call -that- > after the initial object has been created (I know that this should be > a model function which jst returns the firstName + " " + lastName but, > go with me here, I am trying to keep my example simple). > > If there is a 'newRowInDB' flag, then I can say something like; > > def _init(self, id, connection=None, selectResults=None): ''' > > Overload this method if you need to do anything special ''' > > SQLObject._init(self, id, connection, selectResults) if > > self.newRowInDB: self.completeName = self.firstName + " " + > > self.lastName memcache.put("%s_%s" % (self.__class__.__name__, > > self.id)) ._SO_finishCreate() and ._init() are only called on create (INSERT) and not called when the object is fetched from the cache so you don't need a flag here - just set self.completeName in your overridden _init(). Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Dan P. <da...@ag...> - 2009-02-04 22:21:39
|
On Tuesday 03 February 2009, Oleg Broytmann wrote: > On Tue, Feb 03, 2009 at 02:37:16PM -0500, Stef Telford wrote: > > Oleg Broytmann wrote: > > > Because SQLObject needs to generate unique alias names and a global > > > counter is the simplest way to do it. Think about joining a table > > > with itself - you need two different names for the table. > > > > Sorry, perhaps I am being stupid here (always a possability) but, how > > many self joins are we expecting in a query ? I mean to say, isn't > > saying 'selfName = self.__class__.__name__ + "_%d" % int(random())' > > good enough ? > > Are you sure calling random() and int() is really so much faster > than lock/increase counter/unlock? > Are you sure you are optimizing a real stumbling block? I do not think that the issue is one of speed, but one of correctness. Locks plus an incrementing counter is safe and correct. Random numbers are not. While they may work most of the time, they may also fail under given circumstances which is unacceptable. -- Dan |
From: Oleg B. <ph...@ph...> - 2009-02-02 12:11:25
|
On Sun, Feb 01, 2009 at 06:25:07PM -0500, Stef Telford wrote: > Oleg Broytmann wrote: > > My company uses SQLObject solely for desktop applications. > > > Understandable, and this is pretty much how I was thinking SQLObject > was designed/targetted for. This makes perfect sense, but in a web I don't think it is how and what it's designed for. Its original author, Ian Bicking, AFAIK, had been working in a web environment. > framework perhaps not as much. No offense meant ;) No offence taken. :) > >> *) There is writeLocks - surely not required since most > >> frameworks run the entire thread under a transaction anyway. > > > > I don't see how a web transaction is related to multithread-safe > > locks. Whatever a web transaction is, there are a lot of different > > web frameworks, and who guarantees it is thread-safe to touch > > shared variables? > > > well, from what I have seen over the years, most web frameworks that > use an ORM do the following; > start request handling in a thread > create connection to db > do processing > commit connection/rollback capture > end request > > I know that's -very- high level, but, there is never really any > contention in such an environment and, therefore, there is no need for > writeLocks. I am thinking of webware here, probably Turbogears from > what I can recall and .. you get the idea. so, I am suggesting that > the web edition get's -rid- of writeLocks :) Still I don't understand. What do you mean by saying "there is no contention"? If there are threads and there are shared variables - the variables must be protected, right? Have I missed something? There are global counters, e.g. in Alias - how would they fare if different threads would want to increase them? > >> *) cache built in at a fundamental level - most web people would > >> probably feel safer with memcache, currently I don't see anyway > >> to disable this > > > > Probably by writing a Cache class that doesn't cache. > > > yes.. I could do that, but then the 'web' edition would have all these > references to 'cache' laying around that would be confusing for later > people/programmers. I would prefer to keep code clean and simple if at > all possible. Another approach for SQLObject would be to grow an API to disable caching, similar to how one can disable connection pooling. Setting connection.cache to None, or a flag in sqlmeta. But the patch to implement this would be rather big and have to be thoroughly tested. > >> *) no query caching > > > > What is a query in terms of an ORM? > > In -my- usage pattern, it makes a lot of sense. Sometimes a page calls > .count() more than 20 times for the same query/object. Poor design ? > perhaps, however I think that the 'path of least surprise' is that > SQLObject would 'cache' that query and return the cached value since > we are inside the same request/response cycle. Even inside one transaction the result of any .count() could change. But if one is satisfied with caching results of such queries... well, it is up to her/him, I don't mind having an API for such caching. > >> So, is it insanity to propose that perhaps there is a > >> sqlobject-web version? > > > > You are welcome to support one. > > > Well, there's the rub.. I could fork a new version, and then spend > most of my time playing catch-up with the 'official' releases of > SQLObject .. but I don't think that really helps anyone. Is there any > way of making a git branch and then before a release even flinging > across an email/changeset ? I don't know your dev process's involved :D Currently the developers are using Subversion. Worse then, the development process is a bit stalled as I don't have spare time and other developers are not active too. :( Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2009-02-02 19:43:03
|
On Mon, Feb 02, 2009 at 12:38:45PM -0500, Stef Telford wrote: > Oleg Broytmann wrote: > > On Sun, Feb 01, 2009 at 06:25:07PM -0500, Stef Telford wrote: > > No offence taken. :) > > good show. I always get worried about people taking offense over > e-mail. I find that tone is very hard to convey. I know. I did it many time in this mailing list - inadvertently offended people. Different levels of English knowledge, different vocabularies, different understanding... > > Still I don't understand. What do you mean by saying "there is no > > contention"? If there are threads and there are shared variables - > > the variables must be protected, right? Have I missed something? > > There are global counters, e.g. in Alias - how would they fare if > > different threads would want to increase them? > > Well, at the start of every request, a connection is created to the > database. Now, -any- rdbms worth it's salt basically creates an > isolation level so that I can see -my- changes but no one else's > changes Aha, I see. We are talking about different concepts. You are talking about database isolation. I am talking about python shared variables. Those locks in the code protect python variables, not database! Think about that very pool, connection._pool - nobody (i.e., no other thread) should read it even less should write to it until SQLObject finishes manipulating it. The same is true for alias lock. > I don't mind tidying up and putting in > memcache support and master/slave style connections for a web version, > but I obviously don't want to play 'catch up' with every new release > that you fellows do.. hrm. But what was you plan from the beginning? If you were going to have a forked version of SQLObject - wouldn't fork and catching up the only option? You can add some code to the very SQLObject, but if you want to remove some generic code - what is the way? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2009-02-02 21:08:54
|
On Mon, Feb 02, 2009 at 03:07:49PM -0500, Stef Telford wrote: > However, what is the alias lock for ? In the It is a lock that protects shared alias counter in the class sqlbuilder.Alias. > scenario I plan on using SQLObject in, would an alias lock even be > required/needed ? Yes. > alias's will stay 'fixed' at webserver/framework > launch .. no ? Alias counter is increased on every instantiation of Alias(). > Actually, I -had- hoped that SQLObject could have writelocks > enabled/disabled, same for connection pooling (which you tell me it > does, which is great), caching enabled/disabled (the version inside Only row caching needs to be disabled. > SQLObject). The queryCaching I do currently by using python's 'global' > in the servlet , I can't imagine that going into the main SQLObject > (and it probably shouldn't). I could put that in a derived class from > the framework Servlet itself. Ok. > Let me jst say that I have had -bad- previous experience with forks of > projects and libraries. Invariably they start to become more and more > incompatible. I would prefer -not- to get into that situation if > possible. But of course! > Actually, is it possible to get (under subversion) a delta of patches > from one version to another but cherry picking the changes to be made? With some work - that is, save a diff to a text file and edit it. It is just a well-known patch format. > Thank you for your time Oleg, I know my emails may seem to be strange, They are not! You have a need and a vision WRT SO. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2009-02-03 10:42:58
|
On Mon, Feb 02, 2009 at 09:39:06PM -0500, Stef Telford wrote: > Oleg Broytmann wrote: > > On Mon, Feb 02, 2009 at 03:07:49PM -0500, Stef Telford wrote: > >> alias's will stay 'fixed' at webserver/framework > >> launch .. no ? > > > > Alias counter is increased on every instantiation of Alias(). > > Stupid question here .. but .. why ? (sorry, I know that this could be > a large answer but..) Because SQLObject needs to generate unique alias names and a global counter is the simplest way to do it. Think about joining a table with itself - you need two different names for the table. > Also, whilst I am here, a couple more stupid questions, is there any > flag in the SQLObject which tells me if the object was 'created' from > the database or instantiated from the database/cache ? I ask only > because it would be -awfully- handy to have .. I did look at _creating > but, that's not the right item (obviously) There is no and I really fail to see a difference. Why do you want to distinguish them? > Lastly (I promise) I am thinking of over-riding the _SO_finishCreate > in the SQLObject subclass so that all database inserts go to the > master rather than any slave. I would also have to override > _SO_setValue and set (of course) .. any other methods in SQLObject > that would have to be subclassed ? No, these seem to be enough. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |