crawler free download

Showing 4 open source projects for "crawler"

View related business solutions

Software Development Go Clear Filters & Widen Search

Custom VMs From 1 to 96 vCPUs With 99.95% Uptime
General-purpose, compute-optimized, or GPU/TPU-accelerated. Built to your exact specs.

Live migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.

Try Free
Ship Agents Faster
Transform your applications and workflows into powerful agentic systems at global scale.

Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.

Get Started Free
1

Colly

Elegant Scraper and Crawler Framework for Golang

Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling.

Downloads: 0 This Week

Last Update: 2025-03-27
See Project
2

gitcrawl

Local-first GitHub issue and pull request crawler

gitcrawl is a repository crawling and indexing tool associated with the OpenClaw ecosystem, designed to collect, organize, and analyze Git-based project data at scale. The project focuses on automating repository inspection and enabling structured access to source code, commit history, and metadata for research, automation, or AI-assisted workflows. Its architecture likely emphasizes efficient crawling, local indexing, and integration with other OpenClaw tooling for autonomous analysis and...

Downloads: 4 This Week

Last Update: 2 days ago
See Project
3

DHT

BitTorrent DHT Protocol && DHT Spider.

...Set MaxNodes and BlackListMaxSize to fit yourself. DHT aims to implement the standard BitTorrent DHT protocol, not born for crawling the DHT network. NAT Traversal issue. You run the crawler in a local network. It will block ip which looks bad and a good ip may be misjudged.

Downloads: 0 This Week

Last Update: 2023-01-19
See Project
4

go_spider

An awesome Go concurrent Crawler(spider) framework

An awesome Go concurrent Crawler(spider) framework. The crawler is flexible and modular. It can be expanded to an Individualized crawler easily or you can use the default crawl components only. Spider gets a Request in Scheduler that has url to be crawled. Then Downloader downloads the result(html, json, jsonp, text) of the Request. The result is saved in Page for parsing in PageProcesser.

Downloads: 0 This Week

Last Update: 2023-01-27
See Project
Build Securely on Azure with Proven Frameworks
Lay a foundation for success with Tested Reference Architectures developed by Fortinet’s experts. Learn more in this white paper.

Moving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.

Download Now