I have a little question. I successfully used you crawler in a personal site. Now for Website at the company I work for I`m having some issues. The website I'm referring to requires to be logged in. It seems that the crawler doesn't pass the login and only crawls the starting/login page. How can I overcome this problem?
Thank you very much!!
Log in to post a comment.