I tried to use the crawling script to crawl on a domain which I know is active and live. but when I crawled it gaves me HTTP Error 404. The requested resource is not found message.
Is there any specific reason that script is unable to crawl particular web sites or is there any thing to do with robot.txt file.
I also checked robot.txt file but it seems to be not blocking anything
Someone please give me an answer
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I tried to use the crawling script to crawl on a domain which I know is active and live. but when I crawled it gaves me HTTP Error 404. The requested resource is not found message.
Is there any specific reason that script is unable to crawl particular web sites or is there any thing to do with robot.txt file.
I also checked robot.txt file but it seems to be not blocking anything
Someone please give me an answer
Hi!
Could you post the link to that page, than i can run a test.
Thx!