I have modified the example crawler as such: http://pastebin.com/ErtNAbF6
And it is functional.
However. Even though I have set
I still only get back a small crawl such as this: http://pastebin.com/zqWRBTGs
Am I missing something?
You seem to have CSS turned off.
Please don't fill out this field.
You set $crawler->setTrafficLimit(2000 * 1024) in your script.
That's why the crawler stopped, it simply reached the given limit of 2000kb.
(See the summary at the end of the output, it says "Bytes received: 2200335 bytes").
So everything works correct there as i can see.