Robert Bala
-
2012-09-21
- milestone: --> Backlog
I would be very helpful, if the scraping speed (Hitcount) on a webpage could be configurable.
As an idea the httrack (http://www.httrack.com/html/step9_opt2.html) Options-Panel-Limits has many needed functions.
As a minimum the web-harvest should support "max connections/second" to avoid any break down of a webserver/page. Currently webharvest downloads the info with maximum possible connection count per second.