I have installed the spider, but the rules in the robots.txt where ignored.
If I have
User-agent: PHPCrawl
Disallow: /
nothing is crawled. but if I add a file like:
User-agent: PHPCrawl
Disallow: /calendar.php
the crawler spiders the calendar.php file :(
I use php5.4 but I change it to 5.3 for testing.
Maybe the Lines in the robots file will parsed in right way.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
I have installed the spider, but the rules in the robots.txt where ignored.
If I have
User-agent: PHPCrawl
Disallow: /
nothing is crawled. but if I add a file like:
User-agent: PHPCrawl
Disallow: /calendar.php
the crawler spiders the calendar.php file :(
I use php5.4 but I change it to 5.3 for testing.
Maybe the Lines in the robots file will parsed in right way.
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi!
Sorry for the late answer, i overlooked your post.
So, what is your robots.txt look like, like this?
This should the crawler expect to crawl nothing, right?
Or did i misunderstand you?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi I fixed the problem with a link in your buglist
Ah ok.
So which bug was it?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
http://sourceforge.net/p/phpcrawl/bugs/59/