Learn how easy it is to sync an existing GitHub or Google Code repo to a SourceForge project! See Demo


Having problem with some links on webistes...

  • ($status == true)
    this  condition is never executing. I don't know why maybe it is some required i have not done or something else.

    But I corrected this by using time function.
    record time initially and in the loop to get time consumed and if this time greater then $this->socketReadTimeout).
    if ($status == true||($new_time>=$this->socketReadTimeout))


  • Hey,

    doesn't simply $crawler->setStreamTimeout(100) and $crawler->setConnectionTimeout(100) do the trick?
    (Just set it to very high values).
    If you got a timout-problem with that site this should help.

  • Sorry for late reply

    I want to crawl thousand of website to generate data in small time as much as possible, so setting it to $crawler->setStreamTimeout(100) will not do the trick.

  • setStreamTimeout(100) just says that the crawler waits 100 seconds for thr reply of a website until it aborts
    the request.

    It has NOTHING to do with the speed of the crawler. If website/webserver is slow the crawler can't do anything about it.
    It's just a timoit-setting, not a "make it slow"-setting ;)



Cancel   Add attachments