>> Ian Bicking wrote :
>> A process=20
>>generally allows you much more control -- you can kill it off, run it =
>>a different user, you don't get caught by the GIL, etc. This works =
>>with queuing jobs in a database. I was just looking at remoteD lately =
>>well (http://www.neurokode.com/Projects/remoted.php), and it looks =
>>a very easy way to do interprocess queuing without a database.
More direct url to remoted : http://remoted.neurokode.com/
PyLinda (http://www-users.cs.york.ac.uk/~aw/pylinda/) looks more =
experimental but has similar, perhaps more flexible, capabilities.
Conceptually I think tuplespaces like PyLinda would seem to be a good =
fit with web-applications. A cgi-program that coordinated with a =
TupleSpace to hand in requests and pull-out results could be backed by =
any number of servers/processes that did the actual work of processing =
requests and responses. Such a setup would suffer from increased latency =
caused by all the coordination but could scale well and be very hard to =
From: Ian Bicking [mailto:ianb@...]
Sent: Friday, May 21, 2004 4:16 PM
To: John Dickinson
Cc: webware-discuss@...; Ian Sparks
Subject: Re: [Webware-discuss] Task-Queue using Webware...
John Dickinson wrote:
> Some code is below. It is used to take a bunch (1000s) of files and
> merge them into one big file. I hope it helps. The hardest part of
> writing it was debugging the worker thread. If everythong doesn't go
> just right, then you end up with a bunch of threads out there in
> memory taking up cycles.
> I do have one (related) problem with doing this. In another (similar)
> application, I tried to use this technique to prevent server
> timeouts. The problem was that my worker thread was trying to eval a
> huge (~40MB) dictionary. This eval seemed to block everything in the
> entire Webware process. Now if I read the Python documentation
> correctly, that happens because Python's threads are simulated in the
> interpreter and each operation is viewed as atomic. The eval was
> taking a long time and would not allow any other thread in the
> process to do it's thing. Is this correct? If not, what is happening?
> If so, what other way can I do this?
Python uses OS threads, so other threads continue operating. However,=20
there is a global interpreter lock (GIL) which is locked for many=20
operations, and you may have encountered that. It usually doesn't cause =
a problem, but I guess it depends on what you are doing. Imports and=20
some code evaluation, for instance, lock the entire process.
Generally I would suggest putting these workers in their own processes=20
-- if not in deployment, at least for development (to make them easier=20
to debug -- i.e., make it usable as a command-line program). A process=20
generally allows you much more control -- you can kill it off, run it as =
a different user, you don't get caught by the GIL, etc. This works well =
with queuing jobs in a database. I was just looking at remoteD lately as =
well (http://www.neurokode.com/Projects/remoted.php), and it looks like=20
a very easy way to do interprocess queuing without a database.
Ian Bicking / ianb@... / http://blog.ianbicking.org