Work at SourceForge, help us to make it a better place! We have an immediate need for a Support Technician in our San Francisco or Denver office.

Close

crawler dies on large sites

Help
2012-11-18
2013-04-09
  • HI
    I'm running the crawler from the command line and I find it dies on larger sites….
    I have  set_time_limit(0);

     
  • Hi!

    What d you mean with "it dies"? Do you get an php-error? Or does it get stuck (no response anymore) in some way?
    Do you use multiprocessing?
    Ans does it happen on all larger sites over there or just with particular ones?

     


Anonymous


Cancel   Add attachments