ability to set time limit?

[Anonymous]

Is there an ability to set a limit on the amount of time spent crawling?? Crawl as many URLs as you can in say 60 secs?

[Uwe Hunfeld]

Hi!

No, there isn't such an option in phpcrawl (yet).

But you can implement this easily yourself in you extended crawler-class,
just do something like this:

class MyCrawler extends PHPCrawler 
{
  protected $time_limit = 60;
  protected $start_time;

  function handleDocumentInfo($DocInfo) 
  {
    if ($this->start_time == null)
      $this->start_time = time(); // Set start-time

    // ... you normal code

    if (time() - $this->start_time > $this->time_limit)
      return -1; // Abort crawling-process
  }
}

And feel free to open a feature-request for a "setTimeLimit()"-option.

[Anonymous]

Thank You so much!! Thanks for the quick reply and thanks for the code, works perfectly for what I needed.