#292 New vs Old URL for crawler

v1.5
open
nobody
None
1
2013-11-18
2013-11-18
Laurent
No

On the crawl process, would it be possible to set a percentage of new URL to crawl for a crawl run.
For large website, it can take a long time to refetch old URL, and I'd like to keep adding new URL in the index.
I'd like to have the ability to balance between fetch new URL and refetch old URl for a crawl run.

Laurent

Discussion