Re: [Phpwebsite-developers] robots.txt file

[Scott B. <ba...@ba...>]

your right... after I posted I did see that they weren't in those dir...  it
was more for an example.. a poor one.. but an example

I am getting hit alot for the robots.txt file.. only in the root dir...

Scott

----- Original Message -----
From: "Eloi George" <el...@re...>
To: "phpws dev list" <php...@li...>
Sent: Friday, September 05, 2003 5:11 PM
Subject: Re: [Phpwebsite-developers] robots.txt file


> I don't understand why a spider would even be hitting most of those
> directories.  Do have links to them somewhere on your site?
>
>
>
> ----- Original Message -----
> From: "Scott Baer" <ba...@ba...>
> To: "phpws dev list" <php...@li...>
> Sent: Friday, September 05, 2003 3:41 PM
> Subject: [Phpwebsite-developers] robots.txt file
>
>
> > Could we add a robots.txt file to the tar file..
> > Their is a Robots meta option in the Layout manager... but this does not
> > create the text file which is looked for by some spiders.. google for
one.
> > I think for most people, this is put in the root directory of the
domain,
> so
> > it would make since to put it in.
> > I hate getting 404's in my apache log files from spiders crawling my
> site..
> > not that I care really... and in some cases.. It is not required but is
> > listed HTML 4.01 Specification:
> >
>
http://www.w3.org/TR/1999/REC-html401-19991224/appendix/notes.html#h-B.4.1.1
> .
> >
> > Can anyone look in their log files and see if your getting 404 hits for
> the
> > robots.txt file..
> >
> > less access_log |grep robots.txt |wc -l
> > 87
> > in 5 days.. no big deal.. but just something to clean up.
> >
> > Scott
> >
> > #
> > #
> > #    robots.txt file for phpWebsite
> > #
> > #
> >
> > User-agent: *
> > Disallow: /config/
> > Disallow: /admin/
> > Disallow: /convert/
> > Disallow: /images/
> > Disallow: /core/
> > Disallow: /docs/
> > Disallow: /js/
> > Disallow: /lib/
> > Disallow: /mod/
> > Disallow: /manual/
> >
> >
> >
> > -------------------------------------------------------
> > This sf.net email is sponsored by:ThinkGeek
> > Welcome to geek heaven.
> > http://thinkgeek.com/sf
> > _______________________________________________
> > Phpwebsite-developers mailing list
> > Php...@li...
> > https://lists.sourceforge.net/lists/listinfo/phpwebsite-developers
> >
> >
>
>
>
>
> -------------------------------------------------------
> This sf.net email is sponsored by:ThinkGeek
> Welcome to geek heaven.
> http://thinkgeek.com/sf
> _______________________________________________
> Phpwebsite-developers mailing list
> Php...@li...
> https://lists.sourceforge.net/lists/listinfo/phpwebsite-developers
>