Screaming Frog SEO Spider
The Screaming Frog SEO Spider is a website crawler that helps you improve onsite SEO, by extracting data & auditing for common SEO issues. Download & crawl 500 URLs for free, or buy a license to remove the limit & access advanced features. The SEO Spider is a powerful and flexible site crawler, able to crawl both small and very large websites efficiently while allowing you to analyze the results in real-time. It gathers key onsite data to allow SEOs to make informed decisions. Crawl a website instantly and find broken links (404s) and server errors. Bulk export the errors and source URLs to fix, or send to a developer. Find temporary and permanent redirects, identify redirect chains and loops, or upload a list of URLs to audit in a site migration. Analyze page titles and meta descriptions during a crawl and identify those that are too long, short, missing, or duplicated across your site.
Learn more
Hextrakt SEO crawler
Hextrakt is the only desktop crawler that provides a real adaptive asynchronous crawl. It optimizes the crawl speed, taking care of the server and client capacities so that it can crawl efficiently all kind of websites, including big ones. Hextrakt has a nice user-friendly interface, helping the user to explore and segment URLs, focusing on information that matters, to perform relevant technical SEO audits.
Learn more
Semantic Juice
Use capabilities of our web crawler for topical and general web page discovery, open or site specific crawl with powerful domain, URL, and anchor text level rules. Get relevant content from the web, discover new big sites in your niche. Use API for integration with your project. Our crawler is tuned to find topical pages from small set of examples, avoid various spider traps and spam sites, crawl more often more relevant and more topically popular domains, etc. You can define topics, domains, url paths, regular expression, crawling intervals, general, seed, and news crawling modes. Built-in features make our crawlers more efficient as they ignore near duplicate content, spam pages, link farms, and have a real time domain relevancy algoritm which gets you the most relevant content for your topic.
Learn more
CrawlCenter
CrawlCenter is a powerful cloud-based app you can use to find On-Page SEO issues on your site. The app crawls your site on the click of a button and gives you access to 15+ SEO reports for free. CrawlCenter crawls your website and saves the website data in the database. The time taken by the crawler to crawl the site can be few seconds or minutes. Once your site has been crawled, CrawlCenter will open the pages of the report automatically. The SaaS uses the website data to generate 15+ reports. The user must view the reports and filter the data to find On-Page SEO issues on their websites. CrawlCenter makes its users aware of the broken internal and external links. If you use this app, you can get rid of broken link checker plugins/extensions (if you're using them). With CrawlCenter, you can find out the pages on your website with duplicate meta description, title, and keyword tags.
Learn more