Download Latest Version
siteone-crawler-v1.0.9-linux-arm64.tar.gz (28.9 MB)
Get an email when there's a new version of SiteOne Crawler
| Name | Modified | Size | Downloads / Week |
|---|---|---|---|
| Parent folder | |||
| siteone-crawler-v1.0.6-linux-arm64.tar.gz | 2023-12-08 | 26.3 MB | |
| siteone-crawler-v1.0.6-linux-x64.tar.gz | 2023-12-08 | 26.7 MB | |
| siteone-crawler-v1.0.6-macos-arm64.tar.gz | 2023-12-08 | 25.5 MB | |
| siteone-crawler-v1.0.6-macos-x64.tar.gz | 2023-12-08 | 26.2 MB | |
| siteone-crawler-v1.0.6-win-x64.zip | 2023-12-08 | 84.6 MB | |
| README.md | 2023-12-07 | 2.0 kB | |
| v1.0.6 source code.tar.gz | 2023-12-07 | 24.0 MB | |
| v1.0.6 source code.zip | 2023-12-07 | 24.0 MB | |
| Totals: 8 Items | 237.4 MB | 0 | |
The primary change is to fix a bug that in some cases caused asynchronous request queue to get stuck in the last stage of crawling.
Changes
- readme: removed bold links from the intro (it didn't look as good on github as it did in the IDE)
b675873 - readme: improved intro and gif animation with the real output
fd9e2d6 - http auth: for security reasons, we only send auth data to the same 2nd level domain (and possibly subdomains). With HTTP basic auth, the name and password are only base64 encoded and we would send them to foreign domains (which are referred to from the crawled website)
4bc8a7f - html report: increased specificity of the .header class for the header, because this class were also used by the generic class at <td class='header'> in security tab
9d270e8 - html report: improved readability of badge colors in light mode
76c5680 - crawler: moving the decrement of active workers after parsing URLs from the content, where further filling of the queue could occur (for this reason, queue processing could sometimes get stuck in the final stages)
f8f82ab - analysis: do not parse/check empty HTML (it produced unnecessary warning) - it is valid to have content-type: text/html but with connect-lengt: 0 (for example case for 'gtm.js?id=')
436d81b