Re: [Phpwebsite-developers] robots.txt file

[Eloi G. <el...@re...>]

I don't understand why a spider would even be hitting most of those
directories.  Do have links to them somewhere on your site?



----- Original Message -----
From: "Scott Baer" <ba...@ba...>
To: "phpws dev list" <php...@li...>
Sent: Friday, September 05, 2003 3:41 PM
Subject: [Phpwebsite-developers] robots.txt file


> Could we add a robots.txt file to the tar file..
> Their is a Robots meta option in the Layout manager... but this does not
> create the text file which is looked for by some spiders.. google for one.
> I think for most people, this is put in the root directory of the domain,
so
> it would make since to put it in.
> I hate getting 404's in my apache log files from spiders crawling my
site..
> not that I care really... and in some cases.. It is not required but is
> listed HTML 4.01 Specification:
>
http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.4.1.1
.
>
> Can anyone look in their log files and see if your getting 404 hits for
the
> robots.txt file..
>
> less access_log |grep robots.txt |wc -l
> 87
> in 5 days.. no big deal.. but just something to clean up.
>
> Scott
>
> #
> #
> #    robots.txt file for phpWebsite
> #
> #
>
> User-agent: *
> Disallow: /config/
> Disallow: /admin/
> Disallow: /convert/
> Disallow: /images/
> Disallow: /core/
> Disallow: /docs/
> Disallow: /js/
> Disallow: /lib/
> Disallow: /mod/
> Disallow: /manual/
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Phpwebsite-developers mailing list
> Php...@li...
> https://lists.sourceforge.net/lists/listinfo/phpwebsite-developers
>
>