Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo
very nice tools..
when i was trying to get data from https://www.onlinesbi.com.
and crawler will die.
If i set setpagelime(13);
it is working I think there is some problem occuring when crawler goes to this link
Please reply as soon as possible.
You seem to have CSS turned off.
Please don't fill out this field.
($status == true)
this condition is never executing. I don't know why maybe it is some required i have not done or something else.
But I corrected this by using time function.
record time initially and in the loop to get time consumed and if this time greater then $this->socketReadTimeout).
if ($status == true||($new_time>=$this->socketReadTimeout))
doesn't simply $crawler->setStreamTimeout(100) and $crawler->setConnectionTimeout(100) do the trick?
(Just set it to very high values).
If you got a timout-problem with that site this should help.
Sorry for late reply
I want to crawl thousand of website to generate data in small time as much as possible, so setting it to $crawler->setStreamTimeout(100) will not do the trick.
setStreamTimeout(100) just says that the crawler waits 100 seconds for thr reply of a website until it aborts
It has NOTHING to do with the speed of the crawler. If website/webserver is slow the crawler can't do anything about it.
It's just a timoit-setting, not a "make it slow"-setting ;)