Re: [Treebase-devel] Fwd: Treebase Dev problems

[Rutger V. <R....@re...>]

We want search engines to crawl as much of the production server as we can
manage (but remember when the google bot brought it to its knees) but none
of dev and stage - that would only lead to inconsistent search results.
Ideally we would let the bots crawl a site map with the purls so that it is
those that are in the search results (though I wonder whether a bot would
use the purl as the address or whatever that purl forwards to).

On Tuesday, November 15, 2011, Hilmar Lapp <hl...@ne...> wrote:
> I can't think of any good reason why we would want any search engines
crawling any of the dev or staging sites' contents. For production we do,
though, at least so long as it doesn't harm the stability of the site.
>
> -hilmar
>
> Sent with a tap.
>
> On Nov 15, 2011, at 7:50 AM, Mattison Ward <mat...@ne...>
wrote:
>
>> The Nagios monitoring system queries treebase-dev every few minutes to
>> make sure it is up using this query:
>>
>>
http://treebase-dev.nescent.org/treebase-web/search/studySearch.html?query=prism.publicationName=Nature&format=null&recordSchema=null
>>
>>
>> It might be unrelated, but I saw a fair amount of activity from search
>> engines in the web server logs.
>>
>> I can set up a robots.txt file to keep search engines from crawling
>> the dev and staging sites.
>>
>> Would it make sense to keep search engines from crawling any sections
>> of the production site?
>>
>> -Mattison
>>
>> On Mon, Nov 14, 2011 at 5:10 PM, William Piel <wil...@ya...>
wrote:
>>>
>>> Is anyone hitting dev with a lot of queries (e.g. asking for
>>> publication=="Nature") ?  Or is there anything else in these logs that
>>> indicates what might be taking down treebasedev ? (please see the
attached
>>> logs)
>>> Admittedly, it doesn't take much to cause a denial-of-service... But I
was
>>> thinking that it might be in our group, seeing as dev is being hit.
>>> bp
>>>
>>> Begin forwarded message:
>>>
>>> From: Mattison Ward <mat...@ne...>
>>> Date: November 14, 2011 4:12:35 PM EST
>>> To: William Piel <wil...@ya...>, Harry Shyket
>>> <har...@ya...>
>>> Cc: David Palmer <dav...@ne...>
>>> Subject: Treebase Dev problems
>>>
>>> Hi Bill and Harry.
>>>
>>> Starting late last night, the treebasedev tomcat process has been
>>> overloading the server.  I restarted tomcat several times today but
>>> after a few hours the problem occurred again.  No changes have been
>>> made to the system recently and I don't see any deployments from
>>> Hudson recently either.
>>>
>>> I have the treebasedev tomcat service shut down now.
>>>
>>> I have attached the logs for you to review to see if it looks like an
>>> application problem.  I can send them in a different format if you
>>> don't like tgz files.
>>>
>>> Regards,
>>>
>>> Mattison
>>>
>>> ---------- Forwarded message ----------
>>> From: root <ro...@tr...>
>>> Date: Mon, Nov 14, 2011 at 4:07 PM
>>> Subject: treebaselogs
>>> To: mat...@gm...
>>>
>>>
>>> logs
>>>
>>>
>>>
>>> --
>>> Mattison Ward
>>> NESCent at Duke University
>>> 2024 W. Main Street, Suite A200
>>> Durham, NC 27705-4667
>>> 919-668-4585 (desk)
>>> 919-668-4551 (alternate)
>>> 919-668-9198 (fax)
>>>
>>>
>>>
>>>
>>
>>
>>
>> --
>> Mattison Ward
>> NESCent at Duke University
>> 2024 W. Main Street, Suite A200
>> Durham, NC 27705-4667
>> 919-668-4585 (desk)
>> 919-668-4551 (alternate)
>> 919-668-9198 (fax)
>>
>>
------------------------------------------------------------------------------
>> RSA(R) Conference 2012
>> Save $700 by Nov 18
>> Register now
>> http://p.sf.net/sfu/rsa-sfdev2dev1
>> _______________________________________________
>> Treebase-devel mailing list
>> Tre...@li...
>> https://lists.sourceforge.net/lists/listinfo/treebase-devel
>
>
------------------------------------------------------------------------------
> RSA(R) Conference 2012
> Save $700 by Nov 18
> Register now
> http://p.sf.net/sfu/rsa-sfdev2dev1
> _______________________________________________
> Treebase-devel mailing list
> Tre...@li...
> https://lists.sourceforge.net/lists/listinfo/treebase-devel
>

-- 
Dr. Rutger A. Vos
School of Biological Sciences
Philip Lyle Building, Level 4
University of Reading
Reading, RG6 6BX, United Kingdom
Tel: +44 (0) 118 378 7535
http://rutgervos.blogspot.com