I'm using MySQLdb with Python2.2 running under Apache and mod_fastcgi.
I have a Python fcgi script that makes a single connection to the DB, then requests and executes queries (no inserts or updates) from Apache.
It all works, but I recently made a change to a query that caused the query to take 0.3 seconds to run. When I did so, the site slowed from being able to handle 50 requests a second to only being able to handle 3 per second, even though Apache tries to start many instances of the python fcgi script.
An equivalent CGI script written in C that makes a separate connection per instance and connects to the same DB is still able to handle 50 requests a second -- even though each request takes longer to execute. It appears as though each query from MySQLdb waits for all others to complete, even though they are running from separate instances spawned by mod_fcgi.
Is there anything I should be doing to allow them to run in parallel?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
You have one persistent connection that you are sharing? How are you keeping multiple processes from using it? I'm guessing that you aren't on the C version, and have just gotten lucky that you haven't dumped core, or maybe you have and didn't notice.
In any case: If queries are taking a long time, you need more connections. You can use a connection Pool if your app is threaded (I don't really know anythinab out FCGI). See http://dustman.net/andy/python/Pool
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Anonymous
-
2002-09-24
FastCGI allows a single instance of a script to handle multiple web requests. Through the mod_fcgi module, Apache will start an instance of the script when a request comes in. Instead of taking the CGI parameters from the environment and stdin, the script makes an fcgi library call to get the request details, carries out its job, returns the output to Apache and then asks for details about another connection. Whenever an fcgi request comes in, Apache will hand it to a waiting script if there are any. If they are all busy, it will start a new instance up to a configured maximum. The advantage is that there is no process startup or shutdown per request -- each process can live a long time and handle mamy requests in sequence.
In my case, each instance of the script is single threaded and creates its own connection object, but there may be as many as twenty instances running at a time (thus twenty connections). Each instance of the script creates a log entry upon startup so I've confirmed that each script has it's own pid. It appears, though, from the evidence I described earlier, that only one query is able to run at a time.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I'm using MySQLdb with Python2.2 running under Apache and mod_fastcgi.
I have a Python fcgi script that makes a single connection to the DB, then requests and executes queries (no inserts or updates) from Apache.
It all works, but I recently made a change to a query that caused the query to take 0.3 seconds to run. When I did so, the site slowed from being able to handle 50 requests a second to only being able to handle 3 per second, even though Apache tries to start many instances of the python fcgi script.
An equivalent CGI script written in C that makes a separate connection per instance and connects to the same DB is still able to handle 50 requests a second -- even though each request takes longer to execute. It appears as though each query from MySQLdb waits for all others to complete, even though they are running from separate instances spawned by mod_fcgi.
Is there anything I should be doing to allow them to run in parallel?
You have one persistent connection that you are sharing? How are you keeping multiple processes from using it? I'm guessing that you aren't on the C version, and have just gotten lucky that you haven't dumped core, or maybe you have and didn't notice.
In any case: If queries are taking a long time, you need more connections. You can use a connection Pool if your app is threaded (I don't really know anythinab out FCGI). See http://dustman.net/andy/python/Pool
FastCGI allows a single instance of a script to handle multiple web requests. Through the mod_fcgi module, Apache will start an instance of the script when a request comes in. Instead of taking the CGI parameters from the environment and stdin, the script makes an fcgi library call to get the request details, carries out its job, returns the output to Apache and then asks for details about another connection. Whenever an fcgi request comes in, Apache will hand it to a waiting script if there are any. If they are all busy, it will start a new instance up to a configured maximum. The advantage is that there is no process startup or shutdown per request -- each process can live a long time and handle mamy requests in sequence.
In my case, each instance of the script is single threaded and creates its own connection object, but there may be as many as twenty instances running at a time (thus twenty connections). Each instance of the script creates a log entry upon startup so I've confirmed that each script has it's own pid. It appears, though, from the evidence I described earlier, that only one query is able to run at a time.