I am using a bot to crawl some websites. How do I automate changing identity if a required page starts to redirect me to some other page for an excessive amount of traffic coming from "me" via AdvOR? Can I use some sort of error page rules, domain/url masks or maybe check for certain a string in the in the actual HTML coding of redirecting page? And should it suffice to change IP only? Thanks.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Open Discussion" comments posted by this user
Mark all as spam, and block user from posting to "Discussion"
I am using a bot to crawl some websites. How do I automate changing identity if a required page starts to redirect me to some other page for an excessive amount of traffic coming from "me" via AdvOR? Can I use some sort of error page rules, domain/url masks or maybe check for certain a string in the in the actual HTML coding of redirecting page? And should it suffice to change IP only? Thanks.