Thanks for your great project , currently we got a complaint from one of the website that our crawl was too aggressive, is there any way to make crawl send request slow or delay , i know the currently version doesn't support yet , if you can help me to modify to make it works , it will be very helpful for me.
Thank you very much
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi !
Thanks for your great project , currently we got a complaint from one of the website that our crawl was too aggressive, is there any way to make crawl send request slow or delay , i know the currently version doesn't support yet , if you can help me to modify to make it works , it will be very helpful for me.
Thank you very much
View and moderate all "Help" comments posted by this user
Mark all as spam, and block user from posting to "Forum"
Hi!
A very simple solution is just to add a sleep(xyz) or usleep(xyz) at the end of your handleDocumentInfo-method().
Then every process will wait for the given time before it continues with the next URL.
Would this work for you?
Last edit: Anonymous 2013-08-30