Re: [Phpwebsite-developers] robots.txt file

[<ba...@Ba...>]

Your 100% right..  the only hits I get are for the robot.txt in the root
dir..  my example robots.txt file should be very simple.. not like the one
my first e-mail


> I don't understand why a spider would even be hitting most of those
> directories.  Do have links to them somewhere on your site?
>
>
>
> ----- Original Message -----
> From: "Scott Baer" <ba...@ba...>
> To: "phpws dev list" <php...@li...>
> Sent: Friday, September 05, 2003 3:41 PM
> Subject: [Phpwebsite-developers] robots.txt file
>
>
>> Could we add a robots.txt file to the tar file..
>> Their is a Robots meta option in the Layout manager... but this does not
>> create the text file which is looked for by some spiders.. google for
>> one.
>> I think for most people, this is put in the root directory of the
>> domain,
> so
>> it would make since to put it in.
>> I hate getting 404's in my apache log files from spiders crawling my
> site..
>> not that I care really... and in some cases.. It is not required but is
>> listed HTML 4.01 Specification:
>>
> http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.4.1.1
> .
>>
>> Can anyone look in their log files and see if your getting 404 hits for
> the
>> robots.txt file..
>>
>> less access_log |grep robots.txt |wc -l
>> 87
>> in 5 days.. no big deal.. but just something to clean up.
>>
>> Scott
>>
>> #
>> #
>> #    robots.txt file for phpWebsite
>> #
>> #
>>
>> User-agent: *
>> Disallow: /config/
>> Disallow: /admin/
>> Disallow: /convert/
>> Disallow: /images/
>> Disallow: /core/
>> Disallow: /docs/
>> Disallow: /js/
>> Disallow: /lib/
>> Disallow: /mod/
>> Disallow: /manual/
>>
>>
>>
>> -------------------------------------------------------
>> This sf.net email is sponsored by:ThinkGeek
>> Welcome to geek heaven.
>> http://thinkgeek.com/sf
>> _______________________________________________
>> Phpwebsite-developers mailing list
>> Php...@li...
>> https://lists.sourceforge.net/lists/listinfo/phpwebsite-developers
>>
>>
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Phpwebsite-developers mailing list
> Php...@li...
> https://lists.sourceforge.net/lists/listinfo/phpwebsite-developers
>