crawler not crawling outside of site.

Help
Anonymous
2012-12-27
2013-04-09
  • Anonymous - 2012-12-27

    I have modified the example crawler as such: http://pastebin.com/ErtNAbF6

    And it is functional.

    However. Even though I have set

    setFollowMode(0);
    

    I still only get back a small crawl such as this: http://pastebin.com/zqWRBTGs

    Am I missing something?

     
  • Uwe Hunfeld

    Uwe Hunfeld - 2013-01-02

    Hi!

    You set $crawler->setTrafficLimit(2000 * 1024) in your script.
    That's why the crawler stopped, it simply reached the given limit of 2000kb.

    (See the summary at the end of the output, it says "Bytes received: 2200335 bytes").

    So everything works correct there as i can see.

     


Anonymous

Cancel  Add attachments