Limit pages crawled per domian?

  • gyuri79

    gyuri79 - 2014-04-26

    I would like to crawl the web, and limit crawls to x number of pages per url. How can I do that with phpcrawl? I notice I ac only set a global page limit it seems

    Last edit: gyuri79 2014-04-26
  • Comment has been marked as spam. 

    You can see all pending comments posted by this user  here

    Anonymous - 2014-04-29

    Yes, you are right, there is no option to limit the number of crawled pages per domain.

    Maybe you could just crawl a single domain at once with a page limit, collect all links to external domains from it (and store it to a database or something) and then start a new instance of the crawler for every of these domains in a loop?

    But feel free to add a request for a "per domain page-limit" to the list of feature-requestsd! (



Cancel  Add attachments

Get latest updates about Open Source Projects, Conferences and News.

Sign up for the SourceForge newsletter:

No, thanks