I have noted that phpcrawler does not appear to be following the robots.txt files. This is when I set obeyRobots.txt as well as when using the test interface. Am I missing something?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
I have noted that phpcrawler does not appear to be following the robots.txt files. This is when I set obeyRobots.txt as well as when using the test interface. Am I missing something?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
has anyone else had this issue?
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi!
Do you have an example where the robots.txt is not obeyed?
Could you maybe post your setup (including the URL you want to crawl)?
I never encountered such a problem myself.
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
yes, I will post the test interface. I am guessing it must be a php compilation issue. much appreciated.