Re: [Cctools-cchost] bots that ignore robots.txt

[Victor S. <fou...@gm...>]

On 5/8/07, Jon Phillips <jo...@re...> wrote:
> Victor, is the code to do the nasty bot trapping in main ccHost, or is
> it ccMixter and/or is there a way to set it up?
>
> Open Clip Art Library has been getting reamed and had to take emergency
> actions because some bots coming from china are ignoring robots.txt.
>
> Am I remembering correctly that you added a nice fix to ccmixter.org to
> solve this? I would like to add to openclipart.org

Actually for the china bots I took OCA's advice and just used IPTABLES
to block them up front.

The code you're talking about is 5 lines of code that are 'hidden' to
anybody using the site legit. It adds the incoming IP to the deny
section of .htaccess. I don't promote wide use of this technique
because you have to be SURE the links are hidden from legit users
otherwise their IP will give them 403's for the entire site. There's
no way to tell how many legit ccMixter users are in this state since
they are, well, cut off from the site.

VS