Menu

Tree [88f491] master /
 History

HTTPS access


File Date Author Commit
 bin 2023-11-26 jan.reges jan.reges [3cb749] win-x64: upgraded swole-cli.exe.zip with latest...
 build 2023-12-03 jan.reges jan.reges [653187] linux build: changed category to 'Network' and ...
 docs 2023-12-04 jan.reges jan.reges [a84e3f] docs: images and gif animation used in readme
 resources 2023-12-02 jan.reges jan.reges [fd3296] icons: updated all icons to grayscale variant -...
 src 2024-08-24 jan.reges jan.reges [3b4c23] version 1.0.8
 .editorconfig 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .eslintignore 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .eslintrc.cjs 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .gitignore 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .gitmodules 2023-11-25 jan.reges jan.reges [a51a22] git submodule with siteone-crawler: added by co...
 .npmrc 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .prettierignore 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 .prettierrc.yaml 2023-11-20 jan.reges jan.reges [777a39] WIP: partially developed connection of the form...
 LICENSE 2024-08-28 jan.reges jan.reges [88f491] license: change of licensing to MIT
 README.md 2024-08-28 jan.reges jan.reges [88f491] license: change of licensing to MIT
 dev-app-update.yml 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 electron-builder.yml 2023-12-04 jan.reges jan.reges [659df9] electron-builder: unification of the installers...
 electron.vite.config.ts 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 package-lock.json 2024-08-24 jan.reges jan.reges [3b4c23] version 1.0.8
 package.json 2024-08-28 jan.reges jan.reges [88f491] license: change of licensing to MIT
 postcss.config.cjs 2023-11-25 jan.reges jan.reges [30fe0b] tailwind and daisyui integration: integrated by...
 svelte.config.mjs 2023-11-26 jan.reges jan.reges [013cce] WIP: UI redone to DaisyUI, XTerm.js implemented...
 tailwind.config.cjs 2023-11-29 jan.reges jan.reges [ef6a29] basic form: implemented nice user-friendly Basi...
 tsconfig.json 2023-11-19 jan.reges jan.reges [bfdceb] Init commit with first draft version of Crawler...
 tsconfig.node.json 2023-11-20 jan.reges jan.reges [777a39] WIP: partially developed connection of the form...

Read Me

SiteOne Crawler GUI (desktop application)

This is a desktop application to very useful command-line tool SiteOne Crawler (watch video) and is ready for all major platforms - Windows (x64), macOS (x64, arm64) and Linux (x64, arm64).

The goal of this desktop application is to enable comfortable use of SiteOne Crawler even for users without the need to work with the command line.

More info: https://crawler.siteone.io/

Screenshots and video

Animated GIF of the application in action - click to Play on the right when autoplay did not start or watch the commented video.

SiteOne Crawler Demo

Screenshot of the form with full settings. The Basic form contains only the basic settings, and the modifications in it are written in the background to this full form. Here you can also set parameters outside the ranges offered in the basic form.

SiteOne Crawler Full Form

HTML report video

Watch the video that describes the HTML report - one of the main outputs of SiteOne Crawler.

SiteOne Crawler HTML report video

Step 1 - Installation

  1. Go to the releases section.
  2. Download the latest ready version of the installer for your platform (Windows, macOS, Linux) and architecture (x64, arm64).
  3. Install the application.

Step 2 - Usage

  1. Run the installed application - you can find it in your system under the name SiteOne Crawler.
  2. Enter the URL of your website.
  3. Set optional options. Regular users will be fine with the basic configuration form. Advanced users can also use the form for full settings, without limiting the values. The content of these 2 forms is synchronized so you can use them both.
  4. Start crawling.
  5. Wait for the crawling to finish.
  6. On the Result tab, you can open a detailed HTML report or download the report as JSON or TXT. If you have set, you can view the generated offline version of your website or download the sitemap xml/txt files.
  7. All generated content (reports, offline websites, cache, etc.) is stored in the folder SiteOne-Crawler on your desktop.
  8. Take a look at the website crawler.siteone.io and study all the possibilities of use.
  9. Enjoy the tool and send us feedback if you have suggestions for improvement.

For Developers

Important note

This app was written over several evenings with an emphasis on useful functionality and ease of use. Both the functionality and the graphical interface were developed continuously.

The structure of the code is not optimal and the components are not ideally distributed. Sharing state between components is also not optimal.

In the next versions, refactoring will take place with the aim of better dividing the responsibilities of individual components.

If anyone is interested in helping with the development, please open an issue on GitHub. However, I would like to implement other functionalities only after the mentioned refactoring.

Tech Stack

Installation

# checkout repository
git checkout https://github.com/janreges/siteone-crawler-gui.git
cd siteone-crawler-gui

# initialize and download submodule with command-line SiteOne Crawler into src/siteone-crawler
git submodule update --init --recursive

# install dependencies
npm install

Development

# run development server - it will open an Electron app with hot-reloading
$ npm run dev

Build

Note: In the first version, I did not emphasize the treatment of all problems reported by svelte-check. Therefore, to create a build, it is necessary to remove npm run typecheck && from build script in package.json. If anyone wants to participate in the development of this GUI application, I will be happy for help and pull requests.

# NOTICE: builds have to be run on the target platform

# For Windows (x64)
$ npm run build:win-x64

# For macOS (x64)
$ npm run build:mac-x64

# For macOS (arm64)
$ npm run build:mac-arm64

# For Linux (x64)
$ npm run build:linux-x64

# For Linux (arm64)
$ npm run build:linux-arm64

License

This work is licensed under a License: MIT

Want the latest updates on software, tech news, and AI?
Get latest updates about software, tech news, and AI from SourceForge directly in your inbox once a month.