Re: [Semediawiki-user] SD: issues with server load / bing crawler

[Neill M. <ne...@nl...>]

Hi Bernhard.

I have the following in my robots.txt file:

User-agent: *
Disallow: /index.php?
Disallow: /index.php/Help
Disallow: /index.php/MediaWiki
Disallow: /index.php/Special:
Disallow: /index.php/Template
Disallow: /index.php/Template:
Disallow: /index.php/Form
Disallow: /index.php/Form:
Disallow: /index.php/Property
Disallow: /index.php/Property:
Disallow: /skins/
Disallow: /extensions
Disallow: /images

This stops the problem on my servers. Also stops your skins, images and 
extensions folders from being crawled (if you don't already have apache 
rules set for this of course!).

Cheers
Neill.

On 27/05/13 07:57, Krabina Bernhard wrote:
> Dear all,
>
> I am using Semantic Drilldown on many wikis. The largest externally available are
>
> http://www.verwaltungskooperation.at
> http://www.epsa-projects.eu
> http://www.municipal-cooperation.eu
>
> Over time I am experiencing problems with stress on the database from the Drilldown-URL. As far as I can tell, the traffic results from a bing crawler gone wild. As a first counter-measure I enrolled for the bing webmaster tools and told the crawler to slow down generally and to exclude the link to the drilldown interface: (Special:Browse Data and/or Spezial:Daten_durchsuchen)
>
> Does anybody experience similar problems?
>
> Is it just a matter of registering with the webmaster tools or ist there some design issue in SD that could be improved in order to aviod problems with crawlers. In my opinion it does not make sense to crawl this page at all, since it is only doing a view on the content pages that should be crawled. Maybe the special page should have something in it to tell crawlers to get lost?
>
> - Bernhard
>
> ------------------------------------------------------------------------------
> Try New Relic Now & We'll Send You this Cool Shirt
> New Relic is the only SaaS-based application performance monitoring service
> that delivers powerful full stack analytics. Optimize and monitor your
> browser, app, & servers with just a few lines of code. Try New Relic
> and get this awesome Nerd Life shirt! http://p.sf.net/sfu/newrelic_d2d_may
> _______________________________________________
> Semediawiki-user mailing list
> Sem...@li...
> https://lists.sourceforge.net/lists/listinfo/semediawiki-user
>