Abot
Fast and flexible C# framework for building customizable web crawlers
...Abot exposes an event-driven model that enables applications to react to crawling events such as page completion or crawl restrictions. It also provides configuration options that control crawling behavior including concurrency limits, crawl delays, and request parameters. Designed to be lightweight and dependency-free, Abot runs without requiring external services or databases, making it easy to integrate.