crawler dies on large sites

Help
2012-11-18
2013-04-09
  • Nobody/Anonymous

    HI
    I'm running the crawler from the command line and I find it dies on larger sites….
    I have  set_time_limit(0);

     
  • Nobody/Anonymous

    Hi!

    What d you mean with "it dies"? Do you get an php-error? Or does it get stuck (no response anymore) in some way?
    Do you use multiprocessing?
    Ans does it happen on all larger sites over there or just with particular ones?

     


Anonymous

Cancel  Add attachments





Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

JavaScript is required for this form.





No, thanks