SiteOne Crawler is a very useful and easy-to-use tool you'll ♥ as a Dev/DevOps, website owner or consultant. Works on all popular platforms - Windows, macOS, and Linux (x64 and arm64 too). It will crawl your entire website in depth, analyze and report problems, show useful statistics and reports, generate an offline version of the website, generate sitemaps, or send reports via email. Watch a detailed video with a sample report for Astro. build website. This crawler can be used as a command-line tool (see releases and video), or you can use a multi-platform desktop application with a graphical interface (see a video about the app).

Features

  • Very powerful crawler of the entire website reporting useful information about each URL (status code, response time, size, custom headers, titles, etc.)
  • Offers a set of very useful and often necessary features for developers and devops (stress test, warm up cache, localhost testing, etc.)
  • Analyzes all webpages and reports strange or error behaviour and useful statistics (404, redirects, bad practices, etc.)
  • Sends a HTML report to your email addresses with all the information about the crawled website
  • Allows you to export the entire website to offline form, where it is possible to browse the site through local HTML files (without HTTP server) including all images, styles, scripts, fonts, etc.
  • Allows you to generate sitemap.xml and sitemap.txt files with a list of all pages on your website

Project Samples

SiteOne Crawler Screenshot 1

Project Activity

See All Activity >

Categories

Software Development

License

MIT License

Follow SiteOne Crawler

SiteOne Crawler Web Site

Other Useful Business Software
MongoDB Atlas runs apps anywhere Icon
MongoDB Atlas runs apps anywhere

Deploy in 115+ regions with the modern database for every enterprise.

MongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
Start Free
Rate This Project
Login To Rate This Project

User Ratings

5.0 out of 5 stars
★★★★★
★★★★
★★★
★★
★
1
0
0
0
0
ease 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 5 / 5
features 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 5 / 5
design 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 5 / 5
support 1 of 5 2 of 5 3 of 5 4 of 5 5 of 5 5 / 5

User Reviews

Filter Reviews:
All
  • janreges Posted 2024-09-29
    The tool provides a lot of useful information about a website and has a lot of possible uses and options. The ability to export/clone a website is exceptionally good compared to other tools. The primary output is a detailed HTML report (one file) that can be easily shared, including the option to upload it to the cloud using the "--upload" flag and get a secure URL to it. For users who prefer a graphical application, check out crawler.siteone.io for detailed documentation and usage examples.
Read more reviews >

Additional Project Details

Operating Systems

Linux, Mac, Windows

Programming Language

PHP

Related Categories

PHP Software Development Software

Registered

2024-09-13
Report inappropriate content