Menu
▾
▴
Re: [Cctools-cchost] bots that ignore robots.txt
|
From: Jon P. <jo...@re...> - 2007-05-16 00:11:52
|
On Mon, 2007-05-14 at 00:05 +0100, tom whiteley wrote: > Hi, > Had a look at doing this but I can't manage to persuade the xml/tal > stuff to generate javascript at all. Probably obvious if you know how > to do it. > > I'm afraid I'm out of my depth in this framework. I'll try again when > my heads clearer and time allows. > > Tom. I'm cc'ing the cctools list to get more info about this... http://phptal.motion-twin.com/manual/en/ Jon > > > On 5/11/07, Jon Phillips <jo...@re...> wrote: > > On Fri, 2007-05-11 at 12:47 +0100, tom whiteley wrote: > > > Perhaps the tags could use a bit of javascript to generate the target > > > url, so bots couldn't extract it trivially.... > > > > > > What do you think? > > > > > > > > > Tom > > > > Yes, that is a good idea. Right now, the code for tag browsing is in > > cctemplates/tags.xml which creates the interface. > > > > Tom, would you like to take a crack at this? > > > > Jon > > > > > On 5/9/07, Jon Phillips <jo...@re...> wrote: > > > > On Wed, 2007-05-09 at 14:00 -0700, Victor Stone wrote: > > > > > On 5/8/07, Jon Phillips <jo...@re...> wrote: > > > > > > Victor, is the code to do the nasty bot trapping in main ccHost, or is > > > > > > it ccMixter and/or is there a way to set it up? > > > > > > > > > > > > Open Clip Art Library has been getting reamed and had to take emergency > > > > > > actions because some bots coming from china are ignoring robots.txt. > > > > > > > > > > > > Am I remembering correctly that you added a nice fix to ccmixter.org to > > > > > > solve this? I would like to add to openclipart.org > > > > > > > > > > Actually for the china bots I took OCA's advice and just used IPTABLES > > > > > to block them up front. > > > > > > > > > > The code you're talking about is 5 lines of code that are 'hidden' to > > > > > anybody using the site legit. It adds the incoming IP to the deny > > > > > section of .htaccess. I don't promote wide use of this technique > > > > > because you have to be SURE the links are hidden from legit users > > > > > otherwise their IP will give them 403's for the entire site. There's > > > > > no way to tell how many legit ccMixter users are in this state since > > > > > they are, well, cut off from the site. > > > > > > > > > > VS > > > > > > > > >From looking at the logs, it seems that the site is getting reamed from > > > > the multiple tags. > > > > > > > > Damn bots that don't respect robots.txt. Looks like gigablast.com's bots > > > > are not respecting either! > > > > > > > > Hmmm...might have to limit the tag search to 3-4 tags, and/or figure out > > > > a way to make it less juicy for search bots...any suggestions on this > > > > front? > > > > > > > > thx as always vs :) > > > > > > > > Jon > > > > > > > > -- > > > > Jon Phillips > > > > > > > > San Francisco, CA > > > > USA PH 510.499.0894 > > > > jo...@re... > > > > http://www.rejon.org > > > > > > > > MSN, AIM, Yahoo Chat: kidproto > > > > Jabber Chat: re...@gr... > > > > IRC: re...@ir... > > > > > > > > > > > > ------------------------------------------------------------------------- > > > > This SF.net email is sponsored by DB2 Express > > > > Download DB2 Express C - the FREE version of DB2 express and take > > > > control of your XML. No limits. Just data. Click to get it now. > > > > http://sourceforge.net/powerbar/db2/ > > > > _______________________________________________ > > > > Cctools-cchost mailing list > > > > Cct...@li... > > > > https://lists.sourceforge.net/lists/listinfo/cctools-cchost > > > > > > -- > > Jon Phillips > > > > San Francisco, CA > > USA PH 510.499.0894 > > jo...@re... > > http://www.rejon.org > > > > MSN, AIM, Yahoo Chat: kidproto > > Jabber Chat: re...@gr... > > IRC: re...@ir... > > > > -- Jon Phillips San Francisco, CA USA PH 510.499.0894 jo...@re... http://www.rejon.org MSN, AIM, Yahoo Chat: kidproto Jabber Chat: re...@gr... IRC: re...@ir... |
