Thread: [SQLObject] sqlobjects and threads
SQLObject is a Python ORM.
Brought to you by:
ianbicking,
phd
From: Fred C <bs...@gm...> - 2008-04-17 23:03:18
|
I was wandering if there is somewhere an sqlobject 101 for threading environment? I have been googleing for infomation but everything seems confusing. Thanks -fred- |
From: Oleg B. <ph...@ph...> - 2008-04-18 08:13:22
|
On Thu, Apr 17, 2008 at 04:03:08PM -0700, Fred C wrote: > I was wandering if there is somewhere an sqlobject 101 for threading > environment? My personal position is: I avoid threads like a plague. Other than that extreme position - SQLObject is mostly thread-safe, there are people who successfully use it in multithreading TurboGears applications. There is a controversial patch, its author claims without the patch SQLObject hangs, and there are reports that SQLObject hangs with the patch. The patch is now in SQLObject but will be removed in the next release because the last bug report was the patch causes more harm then good. (See why I avoid threads? Threads are evil by nature: everything in programming is about separation of control and responsibility - methods, classes, functions, modules, processes - and threads violate the law by allowing a few processes to share the same process space and memory; that violation leads to deadlocks, race conditions and subtle, hard to debug bugs.) Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Carlos R. <car...@gm...> - 2008-04-20 02:19:19
|
Oleg, That's a rant, it's off topic, and I'm probably posting on the wrong forum, but anyway. I feel your pain, I've spent a good few hours this week debugging threaded programs. At one point I found that I was using the same telnet instance on two threads at the same time. No wonder things were messed up. But it took me two hours to find the root of the problem. But the fact is that today you have to deal with parallelism in one form or another. It can be threads or multiple processes, it's unavoidable. And it will get worse with newer CPUs. Your argument reminds me of a recent post by Guido van Rossum. If I remember it right it was one of the posts regarding Python 3k, someone asked if there were any plans to remove the GIL. Guido said that the someone wrote a patch to remove the GIL. It was very complex and the performance was poor, because of the number of extra checks. In the end it became impossible to keep the patch synchronized with the development on the main Python trunk. Given that and given Guido's opinion regarding threads, the GIL continues to be what it is - a single generic lock for Python instead of a more granular locking mechanism, which would be better for multithreaded programs (at the cost of complexity). Also, note that Google AppEngine uses short lived processes instead of threads. It's a different philosophy, and it permeates everything in Google's design. The hidden assumption is that it's better to scale an application with multiple processes than using threads. But performance may suffer, due to the cost of the synchronization that becomes necessary (note that locking is needed in both cases). Libraries such as SQLObject that keep data cached in memory are heavily affected. There's a potential performance advantage if you use threads with SQLObject because you keep only a single copy of the object in memory for all threads. The difference is noticeable specially for long running processes. Back to the original question, wich asked about a how to for SQLObjects and threads. I believe the same question can be applied if you resort to multiple processes. What's the best way to do it, specially for long running applications? On Fri, Apr 18, 2008 at 5:13 AM, Oleg Broytmann <ph...@ph...> wrote: > On Thu, Apr 17, 2008 at 04:03:08PM -0700, Fred C wrote: > > I was wandering if there is somewhere an sqlobject 101 for threading > > environment? > > My personal position is: I avoid threads like a plague. > > Other than that extreme position - SQLObject is mostly thread-safe, > there are people who successfully use it in multithreading TurboGears > applications. > There is a controversial patch, its author claims without the patch > SQLObject hangs, and there are reports that SQLObject hangs with the > patch. > The patch is now in SQLObject but will be removed in the next release > because the last bug report was the patch causes more harm then good. (See > why I avoid threads? Threads are evil by nature: everything in programming > is about separation of control and responsibility - methods, classes, > functions, modules, processes - and threads violate the law by allowing > a few processes to share the same process space and memory; that violation > leads to deadlocks, race conditions and subtle, hard to debug bugs.) > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ ph...@ph... > Programmers don't die, they just GOSUB without RETURN. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Oleg B. <ph...@ph...> - 2008-04-20 13:19:42
|
On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote: > But the fact is that today you have to deal with parallelism in one form or > another. It can be threads or multiple processes, it's unavoidable. And it > will get worse with newer CPUs. Threads are evil but processes are not - they are completely separated and cannot destroy each other's memory. Processes have less problem (though they still need proper synchronisation). > Your argument reminds me of a recent post by Guido van Rossum. If I remember > it right it was one of the posts regarding Python 3k, someone asked if there > were any plans to remove the GIL. Guido said that the someone wrote a patch > to remove the GIL. It was very complex and the performance was poor, because > of the number of extra checks. In the end it became impossible to keep the > patch synchronized with the development on the main Python trunk. I remember that thread, and Guido resolved that processes are better than threads even on multicore CPUs. Multiprocess model with IPC is the future of Python (and I wholeheartedly agree with this). > Back to the original question, wich asked about a how to for SQLObjects and > threads. I believe the same question can be applied if you resort to > multiple processes. What's the best way to do it, specially for long running > applications? What problems - except of caching - do you foresee? Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Oleg B. <ph...@ph...> - 2008-04-20 15:07:55
|
On Sun, Apr 20, 2008 at 05:19:40PM +0400, Oleg Broytmann wrote: > On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote: > > But the fact is that today you have to deal with parallelism in one form or > > another. It can be threads or multiple processes, it's unavoidable. And it > > will get worse with newer CPUs. > > Threads are evil but processes are not - they are completely separated > and cannot destroy each other's memory. Processes have less problem (though > they still need proper synchronisation). Well, processes may have a shared resource - filesystem. Like a shared memory for threads a shared filesystem requires proper synchronization and locks, and bugs lead to the same old problems - distorted files, deadlocks and race conditions. The real answer for the problem is to avoid filesystem and use databases. No, databases don't have magic powers but they are written by more competent programmers than a general programs and properly implement synchronization and locks. That said, multiprocessing instead of multithreading and databases instead of files is the future of parallel computing. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Carlos R. <car...@gm...> - 2008-04-21 00:45:32
|
The "problem" IMHO is with long running processes. Short-lived processes, in the CGI (and now Google AppEngine) style, should have no problems. Long running processes keep lots of data in memory, and keeping it in sync is very hard. It can be nearly as hard as doing threads right. That was the case I was worried about. But thinking again, you're probably right and this is going to be a non-issue. Few applications today rely on long running processes. Most servers can be implemented efficiently in terms of a "service request" model, using short lived processes. This leaves desktop applications, but that's also changing. It's curious and a bit frustrating. Threads are potentially much more efficient and faster than processes, but are so hard to get right. Perhaps it's just that we got threads because CPUs were slower. Now CPUs are getting fast enough to run entire processes as fast as we expected a threaded program to run. Time to move on? On Sun, Apr 20, 2008 at 10:19 AM, Oleg Broytmann <ph...@ph...> wrote: > On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote: > > But the fact is that today you have to deal with parallelism in one form > or > > another. It can be threads or multiple processes, it's unavoidable. And > it > > will get worse with newer CPUs. > > Threads are evil but processes are not - they are completely separated > and cannot destroy each other's memory. Processes have less problem > (though > they still need proper synchronisation). > > > Your argument reminds me of a recent post by Guido van Rossum. If I > remember > > it right it was one of the posts regarding Python 3k, someone asked if > there > > were any plans to remove the GIL. Guido said that the someone wrote a > patch > > to remove the GIL. It was very complex and the performance was poor, > because > > of the number of extra checks. In the end it became impossible to keep > the > > patch synchronized with the development on the main Python trunk. > > I remember that thread, and Guido resolved that processes are better > than threads even on multicore CPUs. Multiprocess model with IPC is the > future of Python (and I wholeheartedly agree with this). > > > Back to the original question, wich asked about a how to for SQLObjects > and > > threads. I believe the same question can be applied if you resort to > > multiple processes. What's the best way to do it, specially for long > running > > applications? > > What problems - except of caching - do you foresee? > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ ph...@ph... > Programmers don't die, they just GOSUB without RETURN. > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Oleg B. <ph...@ph...> - 2008-04-21 01:03:52
|
On Sun, Apr 20, 2008 at 09:45:28PM -0300, Carlos Ribeiro wrote: > Threads are potentially much more > efficient and faster than processes, but are so hard to get right. They are faster at the cost of fragility. With the speed of hardware constantly going up the cost becomes too big. > Time to move on? Absolutely! Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Frank B. <fb...@fo...> - 2008-04-21 17:39:12
|
Hallo, Oleg Broytmann hat gesagt: // Oleg Broytmann wrote: > On Sat, Apr 19, 2008 at 11:19:14PM -0300, Carlos Ribeiro wrote: > > But the fact is that today you have to deal with parallelism in one form or > > another. It can be threads or multiple processes, it's unavoidable. And it > > will get worse with newer CPUs. > > Threads are evil but processes are not - they are completely separated > and cannot destroy each other's memory. Processes have less problem (though > they still need proper synchronisation). As I'm recently doing more Lua than Python programming: the Lua community is traditionally very sceptical of threads. The alternative approach generally taken in Lua are Coroutines, which found their way into Python some years ago as well ("yield", generators, iterators etc.), but only in a limited way compared to Lua's more powerful and flexible asymmetrical coroutines. An interesting paper in this regard is this: http://www.inf.puc-rio.br/~roberto/docs/corosblp.pdf Coroutines are more lightweight than processes and don't need special synchronisation efforts as only one Coroutine is running at any given time. Ciao -- Frank Barknecht _ ______footils.org__ |
From: Oleg B. <ph...@ph...> - 2008-04-21 17:48:57
|
On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote: > As I'm recently doing more Lua than Python programming: the Lua > community is traditionally very sceptical of threads. The alternative > approach generally taken in Lua are Coroutines, which found their way > into Python some years ago as well ("yield", generators, iterators > etc.), but only in a limited way compared to Lua's more powerful and > flexible asymmetrical coroutines. An interesting paper in this regard > is this: http://www.inf.puc-rio.br/~roberto/docs/corosblp.pdf > > Coroutines are more lightweight than processes and don't need special > synchronisation efforts as only one Coroutine is running at any given > time. http://pypi.python.org/pypi/cogen/ Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Frank B. <fb...@fo...> - 2008-04-21 22:06:47
|
Hallo, Oleg Broytmann hat gesagt: // Oleg Broytmann wrote: > On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote: > > Coroutines are more lightweight than processes and don't need special > > synchronisation efforts as only one Coroutine is running at any given > > time. > > http://pypi.python.org/pypi/cogen/ Yep, that's what I mean. It's often easier and saver to program with coroutines that with threads. In Lua COPAS would be an equivalent to Cogen: http://www.keplerproject.org/copas/ Ciao -- Frank Barknecht |
From: Carlos R. <car...@gm...> - 2008-04-26 10:14:27
|
Coroutines are safer for two reasons. The first is that synchronization is in the programmer's head, while with threads and processes synchronization is done by the operating system. With coroutines you have no deadlocks or race conditions to worry about because you get to explicitly choose your synchronization points. But the main point is that coroutines are written for a single processor world. They emulate parallelism inside a single process. That's what really make them safer. But this is not helpful in a world of multiple cores. One still have to choose between threads and processes for large scale multiprocessing. On Mon, Apr 21, 2008 at 7:06 PM, Frank Barknecht <fb...@fo...> wrote: > Hallo, > Oleg Broytmann hat gesagt: // Oleg Broytmann wrote: > > > On Mon, Apr 21, 2008 at 07:38:55PM +0200, Frank Barknecht wrote: > > > Coroutines are more lightweight than processes and don't need special > > > synchronisation efforts as only one Coroutine is running at any given > > > time. > > > > http://pypi.python.org/pypi/cogen/ > > Yep, that's what I mean. It's often easier and saver to program with > coroutines that with threads. > > In Lua COPAS would be an equivalent to Cogen: > http://www.keplerproject.org/copas/ > > Ciao > -- > Frank Barknecht > > ------------------------------------------------------------------------- > This SF.net email is sponsored by the 2008 JavaOne(SM) Conference > Don't miss this year's exciting event. There's still time to save $100. > Use priority code J8TL2D2. > > http://ad.doubleclick.net/clk;198757673;13503038;p?http://java.sun.com/javaone > _______________________________________________ > sqlobject-discuss mailing list > sql...@li... > https://lists.sourceforge.net/lists/listinfo/sqlobject-discuss > -- Carlos Ribeiro Consultoria em Projetos blog: http://rascunhosrotos.blogspot.com blog: http://pythonnotes.blogspot.com mail: car...@gm... mail: car...@ya... |
From: Oleg B. <ph...@ph...> - 2008-04-26 17:02:47
|
Interview with Donald Knuth: http://www.informit.com/articles/printerfriendly.aspx?p=1193856&rll=1 Definitely very interesting by itself, but in the context of the discussion the most interesting part is about parallel computing, multithreading and multicore hardware. Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |
From: Colin J. W. <cj...@nc...> - 2008-04-27 02:34:09
|
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"> <html> <head> <meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type"> </head> <body bgcolor="#ffffff" text="#000000"> Oleg Broytmann wrote: <blockquote cite="mid:200...@ph..." type="cite"> <pre wrap="">Interview with Donald Knuth: <a class="moz-txt-link-freetext" href="http://www.informit.com/articles/printerfriendly.aspx?p=1193856&rll=1">http://www.informit.com/articles/printerfriendly.aspx?p=1193856&rll=1</a> Definitely very interesting by itself, but in the context of the discussion the most interesting part is about parallel computing, multithreading and multicore hardware. Oleg. </pre> </blockquote> <font face="Times New Roman, Times, serif">Could someone please remove my name from the mailing list?<br> <br> Thanks,<br> <br> Colin W.<br> </font> </body> </html> |
From: Oleg B. <ph...@ph...> - 2008-04-27 05:22:39
|
On Sat, Apr 26, 2008 at 10:33:45PM -0400, Colin J. Williams wrote: > <font face="Times New Roman, Times, serif">Could someone please remove > my name from the mailing list?<br> You can always unsubscribe here: http://lists.sourceforge.net/mailman/listinfo/sqlobject-discuss Oleg. -- Oleg Broytmann http://phd.pp.ru/ ph...@ph... Programmers don't die, they just GOSUB without RETURN. |