Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling. Sync/async/parallel scraping. Distributed scraping. Caching, automatic encoding of non-unicode responses. Robots.txt support. Google App Engine support.

Features

  • Fast (>1k request/sec on a single core)
  • Manages request delays and maximum concurrency per domain
  • Automatic cookie and session handling
  • Sync/async/parallel scraping
  • Distributed scraping
  • Automatic encoding of non-unicode responses

Project Samples

Project Activity

See All Activity >

Categories

Frameworks

License

Apache License V2.0

Follow Colly

Colly Web Site

Other Useful Business Software
AI-generated apps that pass security review Icon
AI-generated apps that pass security review

Stop waiting on engineering. Build production-ready internal tools with AI—on your company data, in your cloud.

Retool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
Try Retool free
Rate This Project
Login To Rate This Project

User Reviews

Be the first to post a review of Colly!

Additional Project Details

Programming Language

Go

Related Categories

Go Frameworks

Registered

2022-11-16