What d you mean with "it dies"? Do you get an php-error? Or does it get stuck (no response anymore) in some way?
Do you use multiprocessing?
Ans does it happen on all larger sites over there or just with particular ones?
If you would like to refer to this comment somewhere else in this project, copy and paste the following link:
HI
I'm running the crawler from the command line and I find it dies on larger sites….
I have set_time_limit(0);
Hi!
What d you mean with "it dies"? Do you get an php-error? Or does it get stuck (no response anymore) in some way?
Do you use multiprocessing?
Ans does it happen on all larger sites over there or just with particular ones?