Hello,
nice project very comperencive i just have two question
does the spider respect the /robots.txt file , and does the rate of the request can be changed somehow ?
thanks
darkeye
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
No, unfortunaltelly not.
Web-Harvest is not real web spider, it is rether web scraper tool.
Example given on the website that collects pages of the website is just illustration of its power.
Vladimir.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
Hello,
nice project very comperencive i just have two question
does the spider respect the /robots.txt file , and does the rate of the request can be changed somehow ?
thanks
darkeye
No, unfortunaltelly not.
Web-Harvest is not real web spider, it is rether web scraper tool.
Example given on the website that collects pages of the website is just illustration of its power.
Vladimir.
too bad :(
thanks for the replay though.