Colly

Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling. Sync/async/parallel scraping. Distributed scraping. Caching, automatic encoding of non-unicode responses. Robots.txt support. Google App Engine support.

Features

Fast (>1k request/sec on a single core)
Manages request delays and maximum concurrency per domain
Automatic cookie and session handling
Sync/async/parallel scraping
Distributed scraping
Automatic encoding of non-unicode responses

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow Colly

Colly Web Site

Other Useful Business Software

Ship Agents Faster

Transform your applications and workflows into powerful agentic systems at global scale.

Gemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.

Get Started Free

Rate This Project

User Reviews

Be the first to post a review of Colly!

Additional Project Details

Programming Language

Related Categories

Go Frameworks

Registered

2022-11-16

Similar Business Software

Koa

Koa is a new web framework designed by the team behind Express, which aims to be a smaller, more expressive, and more robust foundation for web applications and APIs. By leveraging async functions, Koa allows you to ditch callbacks and greatly increase error handling. Koa does not bundle any...

See Software
Echo

High-performance, extensible, minimalist Go web framework. Highly optimized HTTP router with zero dynamic memory allocation which smartly prioritizes routes. Build robust and scalable RESTful API, easily organized into groups. Automatically install TLS certificates from Let's Encrypt. HTTP/2...

See Software
Titanium SDK

Write in JavaScript, and run natively everywhere. Titanium lets you develop cross-platform native mobile applications and build great mobile experiences using JavaScript. Access your application's hundreds of native UI and non-visual components (such as networks and media APIs). Easily include...

See Software
Backbone.js

Backbone.js gives structure to web applications by providing models with key-value binding and custom events, collections with a rich API of enumerable functions, views with declarative event handling, and connects it all to your existing API over a RESTful JSON interface. When working on a web...

See Software
Swoole

Powering the next-generation microservices and applications. Build high-performance, scalable, concurrent TCP, UDP, Unix Socket, HTTP, GRPC services with PHP and easy-to-use coroutine, fibers API. Write your next scalable async application with PHP coroutines and fibers. Compared with other...

See Software
Laravel

Laravel is a web application framework with expressive, elegant syntax. We’ve already laid the foundation — freeing you to create without sweating the small things. We love clean code just as much as you do. Simple, elegant syntax puts amazing functionality at your fingertips. Every feature has...

See Software