[mxbb-commits] core robots.txt,1.1,1.2

[Florin C B. <ory...@us...>]

Update of /cvsroot/mxbb/core
In directory sfp-cvsdas-4.v30.ch3.sourceforge.com:/tmp/cvs-serv24001

Modified Files:
	robots.txt 
Log Message:
robots
Committed on the Free edition of March Hare Software CVSNT Server.
Upgrade to CVS Suite for more features and support:
http://march-hare.com/cvsnt/

Index: robots.txt
===================================================================
RCS file: /cvsroot/mxbb/core/robots.txt,v
retrieving revision 1.1
retrieving revision 1.2
diff -C2 -d -r1.1 -r1.2
*** robots.txt	28 Sep 2009 11:24:28 -0000	1.1
--- robots.txt	27 Apr 2010 16:14:59 -0000	1.2
***************
*** 1,19 ****
! User-agent: msnbot
! User-agent: googlebot
! User-agent: yahoo! slurp
! User-agent: Slurp
! User-agent: lycosbot
! Disallow: /admin
! Disallow: /cache
! Disallow: /config
! Disallow: /include
! Disallow: /install
! Disallow: /modules
  
  User-agent: *
! Disallow: /admin
! Disallow: /cache
! Disallow: /config
! Disallow: /include
! Disallow: /install
! Disallow: /modules
\ No newline at end of file
--- 1,61 ----
! User-agent: * 
! Disallow: /admin/
! Disallow: /cache/
! Disallow: /images/
! Disallow: /includes/
! Disallow: /language/
! Disallow: /modules/
! Disallow: /templates/
! User-Agent: wget
! Disallow: /
! User-Agent: sitecheck.internetseer.com
! Disallow: /
! User-Agent: grub-client
! Disallow: /
! User-Agent: NPBot
! Disallow: /
! User-Agent: WebReaper
! Disallow: /
! User-Agent: Teleport
! Disallow: /
! User-Agent: TeleportPro
! Disallow: /
! User-Agent: MSIECrawler
! Disallow: /
! User-Agent: SiteSnagger
! Disallow: /
! User-Agent: WebStripper
! Disallow: /
! User-Agent: WebCopier
! Disallow: /
! User-Agent: Fetch
! Disallow: /
! User-Agent: Offline Explorer
! Disallow: /
! User-Agent: WebZIP
! Disallow: /
! User-Agent: linko
! Disallow: /
! User-Agent: HTTrack
! Disallow: /
! User-Agent: Microsoft.URL.Control
! Disallow: /
! User-Agent: Xenu
! Disallow: /
! User-Agent: larbin
! Disallow: /
! User-Agent: libwww
! Disallow: /
! User-Agent: ZyBORG
! Disallow: /
! User-Agent: Download Ninja
! Disallow: /
! User-Agent: UbiCrawler
! Disallow: /
! User-Agent: DOC
! Disallow: /
! User-Agent: Zao
! Disallow: /
  
  User-agent: *
! Crawl-delay: 10