I'm running the crawler from the command line and I find it dies on larger sites….
I have set_time_limit(0);
You seem to have CSS turned off.
Please don't fill out this field.
What d you mean with "it dies"? Do you get an php-error? Or does it get stuck (no response anymore) in some way?
Do you use multiprocessing?
Ans does it happen on all larger sites over there or just with particular ones?