Webware for Python

Leigh,

What's the CPU load like when the reporting page is being accessed?  I 
have something similar in one of my projects and had to introduce 
caching for the reporting pages due to the way it would hammer the CPU 
(99% usage for upwards of 10 minutes).  Luckily I have the application 
running on a dual processor machine, so the second processor would be 
available to handle further incoming requests.

I'm still running Webware 0.8.1 myself and have found the only 
limitation to handling simultaneous and long running requests is CPU 
availability.

Hope this helps,

Aaron



Leigh Dyer wrote:

> Hi,
>
> I've been using WebWare in a low-volume production environment for a 
> while now, but as the load has increased I've started to notice some 
> crack forming. Specifically, one issues that causes trouble is that 
> when someone accesses a reporting page on one site, which can take a 
> minute or two to run, other sites seem to be largely unaccessible 
> until the original request is complete.
>
> Is this a known issue, or something odd with my particular setup? -- 
> I'm assuming the latter for now, given the lack of similar complaints. 
> The server in question is running WebWare 0.8.1 under Apache 2.0 
> (using the mod_webkit2 connector) on Debian Sarge, using Python 2.3. 
> The apps are connecting to a SQL Server database using the Object 
> Craft Sybase module. If there's any other info I can provide please 
> let me know.
>
> Thanks
> Leigh
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting 
> language
> that extends applications into web and mobile media. Attend the live 
> webcast
> and join the prime developer group breaking into this new coding 
> territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>
>

Leigh Dyer wrote:
> ... one issues that causes trouble is that when
> someone accesses a reporting page on one site, which can take a minute 
> or two to run, other sites seem to be largely unaccessible until the 
> original request is complete.
> 
> Is this a known issue, or something odd with my particular setup? ...
 > The apps are connecting to a SQL Server database using the Object Craft
 > Sybase module.

Does the Sybase module support connection pooling or did you try with 
Miscutils.DBPool? Maybe the module is using locks to ensure thread 
safety, and these locks block other Webware threads trying to connect to 
the database.

-- Christoph

On Sat, 2006-02-25 at 17:57 +0100, Christoph Zwerschke wrote:
> Leigh Dyer wrote:
> > ... one issues that causes trouble is that when
> > someone accesses a reporting page on one site, which can take a minute 
> > or two to run, other sites seem to be largely unaccessible until the 
> > original request is complete.
> > 
> > Is this a known issue, or something odd with my particular setup? ...
>  > The apps are connecting to a SQL Server database using the Object Craft
>  > Sybase module.
> 
> Does the Sybase module support connection pooling or did you try with 
> Miscutils.DBPool? Maybe the module is using locks to ensure thread 
> safety, and these locks block other Webware threads trying to connect to 
> the database.

Thanks for all the suggestions -- I think I've got some ideas to work
with now.

I've been using DBPool, though recently I've been looking at moving to
PooledDB from DBUtils. I'll have to check in to the CPU load, but I
don't think it would be overly high (the reports are database-limited),
and there's a second CPU to help out as well, so it sounds like the
Sybase driver might be the problem. We have other apps that use
PostgreSQL (which I'm using with the psycopg connector and DBPool) --
I'll see if I can reproduce the problem with some kind of long-running
task that uses that database instead.

Thanks
Leigh


> 
> -- Christoph
> 
> 
> -------------------------------------------------------
> This SF.Net email is sponsored by xPML, a groundbreaking scripting language
> that extends applications into web and mobile media. Attend the live webcast
> and join the prime developer group breaking into this new coding territory!
> http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
> _______________________________________________
> Webware-discuss mailing list
> Webware-discuss@...
> https://lists.sourceforge.net/lists/listinfo/webware-discuss
>

Leigh Dyer wrote:
> We have other apps that use PostgreSQL (which I'm using with the psycopg
 > connector and DBPool) -- I'll see if I can reproduce the problem with
 > some kind of long-running task that uses that database instead.

I am using PostgreSQL, PygreSQL and DBUtils and do not see such 
problems, not even on slow PCs with only one CPU. I.e. I can have one 
lonq query running in one browser connection, and other browser 
connections will not be affected (except things are getting slower of 
course if several such queries are running at the same time).

If you're using psycopg already, I think DBPool or DBUtils is not needed 
because psycopg has its own connection pooling (haven't tested it).

-- Christoph

Christoph Zwerschke wrote:
> Leigh Dyer wrote:
>> We have other apps that use PostgreSQL (which I'm using with the psycopg
>  > connector and DBPool) -- I'll see if I can reproduce the problem with
>  > some kind of long-running task that uses that database instead.
> 
> I am using PostgreSQL, PygreSQL and DBUtils and do not see such 
> problems, not even on slow PCs with only one CPU. I.e. I can have one 
> lonq query running in one browser connection, and other browser 
> connections will not be affected (except things are getting slower of 
> course if several such queries are running at the same time).
> 
That's good to hear -- hopefully I'll get similar behaviour without the 
Sybase module. The Sybase module has always been a bit of a pain to work 
with, but I've just discovered a newer MSSQL module (I'm using the 
Sybase module with SQL Server) which seems better designed to work with 
FreeTDS, and is hopefully more mature in other ways as well. I plan to 
test it shortly to see if it avoids these issues, and I'll let you know 
how it all goes.

Thanks
Leigh

Christoph Zwerschke wrote:
> Leigh Dyer wrote:
>> We have other apps that use PostgreSQL (which I'm using with the psycopg
>  > connector and DBPool) -- I'll see if I can reproduce the problem with
>  > some kind of long-running task that uses that database instead.
> 
> I am using PostgreSQL, PygreSQL and DBUtils and do not see such 
> problems, not even on slow PCs with only one CPU. I.e. I can have one 
> lonq query running in one browser connection, and other browser 
> connections will not be affected (except things are getting slower of 
> course if several such queries are running at the same time).

It looks like the Sybase module was definitely the problem -- I've just 
run some preliminary tests with the PyMSSQL module (pymssql.sf.net), and 
  I've had no trouble accessing other pages while one of my report pages 
are running.

The Sybase module was always a bit of a thorn in my side (this isn't the 
only problem I've had with it), so I'll be very glad to replace it if 
this newer module works well in extended testing. Thanks to you all for 
your help with this.

Thanks
Leigh