[Phpwebsite-developers] robots.txt file

[Scott B. <ba...@ba...>]

Could we add a robots.txt file to the tar file..
Their is a Robots meta option in the Layout manager... but this does not
create the text file which is looked for by some spiders.. google for one.
I think for most people, this is put in the root directory of the domain, so
it would make since to put it in.
I hate getting 404's in my apache log files from spiders crawling my site..
not that I care really... and in some cases.. It is not required but is
listed HTML 4.01 Specification:
http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.4.1.1.

Can anyone look in their log files and see if your getting 404 hits for the
robots.txt file..

less access_log |grep robots.txt |wc -l
87
in 5 days.. no big deal.. but just something to clean up.

Scott

#
#
#    robots.txt file for phpWebsite
#
#

User-agent: *
Disallow: /config/
Disallow: /admin/
Disallow: /convert/
Disallow: /images/
Disallow: /core/
Disallow: /docs/
Disallow: /js/
Disallow: /lib/
Disallow: /mod/
Disallow: /manual/