Re: [Treebase-devel] Treebase Dev problems

[Shyket, H. <har...@ya...>]

The exceptions from the logs seem to be thrown already (just making them look a little cleaner for the user).  Do you have a monitor to see what traffic comes in/how much memory is in use?

Harry Shyket
Digital Media Specialist
Yale University Peabody Museum
ph. 203-436-9428
har...@ya...

From: Mattison Ward [mailto:mat...@ne...]
Sent: Thursday, November 17, 2011 9:12 AM
To: William Piel
Cc: TreeBASE devel
Subject: Re: [Treebase-devel] Treebase Dev problems

If/When we have problems today, I will check the logs to see if there are a large number of phylows requests at the time of the problem.

I would be curious to see if fixing the uncaught exceptions takes care of most of the overloading issues.

Apache has a module that can limit the number of requests per second to specific URLs. For example, requests to /treebase-web/phylows/ could be limited to 10 requests per second while requests to any other URL could be unlimited.  That would be another option to control overloading by API requests if we can isolate the URLs.

-Mattison

On Thu, Nov 17, 2011 at 12:01 AM, William Piel <wil...@ya...<mailto:wil...@ya...>> wrote:

Hi all,

So we continue to get hammered by, I think, someone who is making heavy use of the API. Maybe Mattison can better identify the source of this -- but I don't think it's just higher UI traffic or robot activities because Google Analytics is not showing a jump in activity. While logs show some uncaught exceptions (and Harry is fixing some of these) these are probably not actually causing the slowdown. Rather, objects in memory are probably not being disposed of fast enough and the database is clogging up (direct queries against postgresql are also very slow).

We need a solution because when TreeBASE bogs down it becomes too slow to submit data, etc.  In fact, right now (11:50PM) TreeBASE doesn't seem to be responding at all.

Can anyone suggest a solution?

Would this be a possible solution: deploy another VM with Tomcat and another database instance (e.g. take over treebase-stage.nescent.org<http://treebase-stage.nescent.org> for this purpose). The database is refreshed nightly from production; the war is rebuilt whenever production is rebuilt; but the two are sufficiently isolated so that heavy API activity doesn't cripple production. Traffic headed to http:://www.treebase.org<http://www.treebase.org> goes to production. Traffic headed to http://purl.org/phylo/treebase/phylows/ goes to this other instance.

I guess one possible snag is that when a /phylows/ query includes "format=html", it is jumping to the web page environment -- yet we don't want this API machine to serve web pages to people in any way. So we'd want a /phylows/...?format=html to redirect to production instead.

bp








--
Mattison Ward
NESCent at Duke University
2024 W. Main Street, Suite A200
Durham, NC 27705-4667
919-668-4585 (desk)
919-668-4551 (alternate)
919-668-9198 (fax)