Re: [GMOD-devel] [Gmod-gbrowse] Any solutions to the GBrowse memory leak problem?

SourceForge Headquarters 1320 Columbia Street Suite 310 San Diego, CA 92101 +1 (858) 422-6466

Hi,

That was my original bug report. We're using mod_perl 2.0.4 on Apache 2.2.9,
perl 5.10.0.

Consistently on our servers, it looks like gbrowse spawned child processes
periodically never finish. Our typical apache processes run as:

/usr/sbin/apache2 -f /scratch/www-data/apache2/browser/apache2.conf -k start

whereas long running processes that eventually consume multiple gigs of RAM
and >50% CPU show up in the process list as:

/scratch/www-data -f /scratch/www-data/apache2/browser/apache2.conf -k start

When I run strace against one of these runaway processes, this occasionally
reports a constant stream of mysql queries against our
Bio::SeqFeature::Store database running select statements for subfeatures.
We've had to use Apache:Resource in our perl.conf to kill any process
consuming more than 5 Gb of RAM:
   PerlSetEnv PERL_RLIMIT_DATA 1024:2048
   PerlSetEnv PERL_RLIMIT_AS   5000:6000
   PerlChildInitHandler Apache2::Resource
and also have a cron job to restart/cycle Apache twice a day, but these are
clearly bandaids over a larger problem.

I am in the process of upgrading to BioPerl 1.6.9 and the latest Gbrowse 2
build on a test machine to see if those cumulative bug fixes help. (We're
still on Gbrowse 2.16).

Also, I've seen a few reports of potential memory leaks in either DBI or
DBD::mysql that could be the culprit, although this will require more
testing.

https://rt.cpan.org/Public/Bug/Display.html?id=67435
https://rt.cpan.org/Public/Bug/Display.html?id=66596

Any and all comments welcome, thanks!

-- 
Richard D. Hayes, Ph.D.
Joint Genome Institute / Lawrence Berkeley National Lab
http://www.phytozome.net

On Tue, Apr 19, 2011 at 10:05 AM, Jayaraman, Pushkala <pja...@mc...>wrote:

> RUN(): POST: http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/action=retrieve_multiple;track_ids
>
>
> =HumanSyntenyBlock;track_ids=RatSyntenyBlock;track_ids=dbSNP_Track;tk_HumanSyntenyBlock=26117e647b7417d62fcfd434df39df4b;tk_RatSyntenyBlock=9b2c1f8a09b0e005fd06638a9645eb78;tk_db
>
> SNP_Track=a653ea6516e414ecf2af259e986b2442;_= at
> /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Bio/Graphics/Browser2/Render.pm
> line 201., referer: http://test.rgd.mc
>
> w.edu/fgb2/gbrowse/rgd_904/
>
> [Tue Apr 19 11:56:46 2011] [warn] [client 141.106.161.52] mod_fcgid:
> stderr: [11552] add_user_tracks() at
> /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-, referer: http://test.rgd
>
> .mcw.edu/fgb2/gbrowse/rgd_904/
>
> [Tue Apr 19 11:56:46 2011] [warn] [client 141.106.161.52] mod_fcgid:
> stderr: thread-multi/Bio/Graphics/Browser2/Render.pm line 209., referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse
>
> /rgd_904/
>
> [Tue Apr 19 11:56:46 2011] [warn] [client 141.106.161.52] mod_fcgid:
> stderr: [11552] testing for asynchronous event() at
> /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-mult
>
> i/Bio/Graphics/Browser2/Render.pm line 214., referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/
>
> [Tue Apr 19 11:56:46 2011] [warn] [client 141.106.161.52] mod_fcgid:
> stderr: [11552] asynchronous exit at
> /usr/lib64/perl5/site_perl/5.8.8/x86_64-linux-thread-multi/Bio/Graphics/
>
> Browser2/Render.pm line 216., referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/
>
> *[Tue Apr 19 11:57:44 2011] [warn] [client 141.106.161.52] mod_fcgid: read
> data timeout in 40 seconds, referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/*
>
> *[Tue Apr 19 11:57:44 2011] [error] [client 141.106.161.52] Premature end
> of script headers: gbrowse, referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/*
>
> *[Tue Apr 19 11:57:53 2011] [warn] [client 141.106.161.52] mod_fcgid: read
> data timeout in 40 seconds, referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/*
>
> *[Tue Apr 19 11:57:53 2011] [error] [client 141.106.161.52] Premature end
> of script headers: gbrowse_img, referer:
> http://test.rgd.mcw.edu/fgb2/gbrowse/rgd_904/***
>
>
>
> I see these errors in my error logs.. I see a “read data timeout” .. how do
> I counter that?
>
>
>
> Pushkala Jayaraman
>
> Programmer/Analyst
>
> Rat Genome Database
>
> Human and Molecular Genetics Center
>
> Medical College of Wisconsin
>
> Email: pja...@mc...
>
> Work: 414-955-2229
>
> www.rgd.mcw.edu
>
>
>
> *From:* Jayaraman, Pushkala [mailto:pja...@mc...]
> *Sent:* Tuesday, April 19, 2011 11:34 AM
> *To:* gmo...@li...; gmo...@li...
> *Subject:* [Gmod-gbrowse] Any colutions to the GBrowse memory leak
> problem?
>
>
>
> Hello,
>
> I am referring to this particular issue that we seem to be experiencing as
> well…
>
> http://comments.gmane.org/gmane.science.biology.gmod.gbrowse/7572
>
>
>
> Our Gbrowse process simply hangs even if we have just three tracks open for
> a range of 1MB. Reloading the page seems to make it okay and everything
> renders quickly..
>
> Our Gbrowse process is 2.15.
>
>
>
> The issue ( follow link) reports that it is some sort of memory leak in
> gbrowse..
>
>
>
> Would you guys out there know any solution to this problem.. or a fix that
> we might have to integrate?
>
>
>
>
>
> Pushkala Jayaraman
>
> Programmer/Analyst
>
> Rat Genome Database
>
> Human and Molecular Genetics Center
>
> Medical College of Wisconsin
>
> Email: pja...@mc...
>
> Work: 414-955-2229
>
> www.rgd.mcw.edu
>
>
>
>
> ------------------------------------------------------------------------------
> Benefiting from Server Virtualization: Beyond Initial Workload
> Consolidation -- Increasing the use of server virtualization is a top
> priority.Virtualization can reduce costs, simplify management, and improve
> application availability and disaster protection. Learn more about boosting
> the value of server virtualization. http://p.sf.net/sfu/vmware-sfdev2dev
> _______________________________________________
> Gmod-gbrowse mailing list
> Gmo...@li...
> https://lists.sourceforge.net/lists/listinfo/gmod-gbrowse
>
>