($status == true)
this condition is never executing. I don't know why maybe it is some required i have not done or something else.
But I corrected this by using time function.
record time initially and in the loop to get time consumed and if this time greater then $this->socketReadTimeout).
if ($status == true||($new_time>=$this->socketReadTimeout))
thanks
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
doesn't simply $crawler->setStreamTimeout(100) and $crawler->setConnectionTimeout(100) do the trick?
(Just set it to very high values).
If you got a timout-problem with that site this should help.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
I want to crawl thousand of website to generate data in small time as much as possible, so setting it to $crawler->setStreamTimeout(100) will not do the trick.
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
setStreamTimeout(100) just says that the crawler waits 100 seconds for thr reply of a website until it aborts
the request.
It has NOTHING to do with the speed of the crawler. If website/webserver is slow the crawler can't do anything about it.
It's just a timoit-setting, not a "make it slow"-setting ;)
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
very nice tools..
when i was trying to get data from https://www.onlinesbi.com.
setpagelimit(100);
and crawler will die.
If i set setpagelime(13);
it is working I think there is some problem occuring when crawler goes to this link
"https://www.onlinesbi.com/personal/osbi_etdr_estdr_faq.html"
Please reply as soon as possible.
($status == true)
this condition is never executing. I don't know why maybe it is some required i have not done or something else.
But I corrected this by using time function.
record time initially and in the loop to get time consumed and if this time greater then $this->socketReadTimeout).
if ($status == true||($new_time>=$this->socketReadTimeout))
thanks
Hey,
doesn't simply $crawler->setStreamTimeout(100) and $crawler->setConnectionTimeout(100) do the trick?
(Just set it to very high values).
If you got a timout-problem with that site this should help.
Sorry for late reply
I want to crawl thousand of website to generate data in small time as much as possible, so setting it to $crawler->setStreamTimeout(100) will not do the trick.
setStreamTimeout(100) just says that the crawler waits 100 seconds for thr reply of a website until it aborts
the request.
It has NOTHING to do with the speed of the crawler. If website/webserver is slow the crawler can't do anything about it.
It's just a timoit-setting, not a "make it slow"-setting ;)