Semantic Juice
Use capabilities of our web crawler for topical and general web page discovery, open or site specific crawl with powerful domain, URL, and anchor text level rules. Get relevant content from the web, discover new big sites in your niche. Use API for integration with your project. Our crawler is tuned to find topical pages from small set of examples, avoid various spider traps and spam sites, crawl more often more relevant and more topically popular domains, etc. You can define topics, domains, url paths, regular expression, crawling intervals, general, seed, and news crawling modes. Built-in features make our crawlers more efficient as they ignore near duplicate content, spam pages, link farms, and have a real time domain relevancy algoritm which gets you the most relevant content for your topic.
Learn more
Screaming Frog SEO Spider
The Screaming Frog SEO Spider is a website crawler that helps you improve onsite SEO, by extracting data & auditing for common SEO issues. Download & crawl 500 URLs for free, or buy a license to remove the limit & access advanced features. The SEO Spider is a powerful and flexible site crawler, able to crawl both small and very large websites efficiently while allowing you to analyze the results in real-time. It gathers key onsite data to allow SEOs to make informed decisions. Crawl a website instantly and find broken links (404s) and server errors. Bulk export the errors and source URLs to fix, or send to a developer. Find temporary and permanent redirects, identify redirect chains and loops, or upload a list of URLs to audit in a site migration. Analyze page titles and meta descriptions during a crawl and identify those that are too long, short, missing, or duplicated across your site.
Learn more
SiteGuru
SiteGuru is an on-page SEO tool that checks your website for SEO issues and gives you actionable tips to improve your website.
SiteGuru crawls your site and checks all relevant SEO properties of your site: meta descriptions, page titles, canonical URLs, broken links, page speed and much more.
Export your reports to PDF or CSV, and share your report with your colleagues and developer team.
Learn more
Netpeak Spider
Netpeak Spider is an SEO crawler for a day-to-day SEO audit, fast issue check, comprehensive analysis, and website scraping.
This tool allows you to:
* Spot 100+ issues of your website optimization.
* Check 80+ key on-page SEO parameters.
* Calculate internal PageRank to improve website linking structure.
* Analyze all incoming and outgoing internal links.
* View page source and HTTP headers.
* Generate sitemaps: XML, Image and HTML.
* Adjust Netpeak Spider to your own requirements using crawling modes for the entire website, the URL list or XML Sitemap.
* Set custom rules to crawl either the entire website or its certain part
* Consider indexation instructions (Robots.txt, Meta Robots, X-Robots-Tag, Canonical)
* Perform custom search of source code/text using 4 types of search.
* Avoid duplicate content: Pages, Titles, Meta Descriptions, H1 Headers, etc.
* Spot issues with redirects.
* Overview panel for fast SEO audit with special status codes which show website
Learn more
