[Jcrawler-main] Issues with crawling

[Krishna C. R. B. <cha...@gm...>]

Hello All:We are trying to use Jcrawler to crawl through our site to build
up the cache at regular intervals and we do not want it to run and generate
load.
Is there a way to stop the crawl after it hits all the unique URLs.
Also more importantly can we make it crawl only unique URLs.

Thank you,
KC