I think a small delay while crawling the next url will prevent webmasters from kicking the crawler if the load is too high.
I inserted this in phpcrawler.class.php line 191
sleep(rand(0,3));
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I think a small delay while crawling the next url will prevent webmasters from kicking the crawler if the load is too high.
I inserted this in phpcrawler.class.php line 191
sleep(rand(0,3));
Hi Heiko,
just added your request (for a setDelay()-function or similar) to the list of feature-requests:
https://sourceforge.net/tracker/?func=detail&aid=3500651&group_id=89439&atid=590149
Thanks and Best Regards,
huni.