Search Results for "scrapy-redis"
Sort By:
Showing 193 open source projects for "scrapy-redis"
View related business solutions-
Payroll Services for Small Businesses | QuickBooksEasily pay your team and access powerful tools, employee benefits, and supportive experts with the #1 online payroll service provider. Manage payroll and access HR and employee services in one place. Pay your team automatically once your payroll setup is complete. We'll calculate, file, and pay your payroll taxes automatically.
-
Save hundreds of developer hours with components built for SaaS applications.Whether you want full self-service analytics or simpler multi-tenant security, Qrvey’s embeddable components and scalable data management remove the guess work.
-
1
Scrapy-Redis
Redis-based components for Scrapy
You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3... -
2
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
3
Redis Store
Cache Redis stores for Ruby web frameworks
Redis Store provides a full set of stores (Cache, I18n, Session, HTTP Cache) for modern Ruby frameworks like: Ruby on Rails, Sinatra, Rack, Rack::Cache and I18n. It supports object marshaling, timeouts, single or multiple nodes, and namespaces. -
4
redis-operator
Redis Operator creates/configures/manages high availability redis
Redis Operator creates/configures/manages redis-failovers atop Kubernetes. Kubernetes version: 1.21 or higher Redis version: 6 or higher. Redis operator is being tested against Kubernetes 1.25 1.26 1.27 and redis 6 All dependencies have been vendored, so there's no need to any additional downloads. -
Bright Data - All in One Platform for Proxies and Web ScrapingBright Data offers the highest quality proxies with automated session management, IP rotation, and advanced web unlocking technology. Enjoy reliable, fast performance with easy integration, a user-friendly dashboard, and enterprise-grade scaling. Powered by ethically-sourced residential IPs for seamless web scraping.
-
5
Prometheus Redis Metrics Exporter
Prometheus Exporter for Redis Metrics. Supports Redis 2.x, 3.x, 4.x, 5
Prometheus exporter for Redis metrics. Supports Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x. To have instances in the drop-down as human readable names rather than IPs, it is suggested to use instance relabelling. The Prometheus docs have a very informative article on how multi-target exporters are intended to work. Run the exporter with the command line flag --redis.addr= so it won't try to access the local instance every time the /metrics endpoint is scraped. If authentication is needed... -
6
Node Redis
A high performance Node.js Redis client
This library is a 1 to 1 mapping of the Redis commands. Each Redis command is exposed as a function on the client object. All functions take either an args Array plus optional callback Function or a variable number of individual arguments followed by an optional callback. Node Redis currently doesn't natively support promises (this is coming in v4), however you can wrap the methods you want to use with promises using the built-in Node.js util.promisify method on Node.js >= v8. Care should... -
7
graphql-redis-subscriptions
A graphql subscriptions implementation using redis
This package implements the PubSubEngine Interface from the graphql-subscriptions package and also the new AsyncIterator interface. It allows you to connect your subscriptions manager to a Redis Pub Sub mechanism to support multiple subscription manager instances. As the graphql-subscriptions package is declared as a peer dependency, you might receive a warning about an unmet peer dependency if it's not installed already by one of your other packages. Calling the method asyncIterator... -
8
Spring Data Redis
Provides support to increase developer productivity in Java
Provides support to increase developer productivity in Java when using Redis, a key-value store. Uses familiar Spring concepts such as a template class for core API usage and lightweight repository-style data access. The primary goal of the Spring Data project is to make it easier to build Spring-powered applications that use new data access technologies such as non-relational databases, map-reduce frameworks, and cloud-based data services. Connection package as low-level abstraction across... -
9
DbGate
Database manager for MySQL, PostgreSQL, SQL Server, MongoDB
Database manager for MySQL, PostgreSQL, SQL Server, MongoDB, SQLite and others. Runs under Windows, Linux, Mac or as web application. DbGate is a cross-platform database manager. It's designed to be simple to use and effective when working with more databases simultaneously. But there are also many advanced features like schema compare, visual query designer, chart visualization or batch export and import. Mongo JavaScript editor, execute Mongo script (with NodeJs syntax). Redis tree view... -
Start building the next generation of GenAI apps todayMongoDB Atlas is a fully-managed developer data platform built by developers, for developers. With tight integration to Google Cloud services such as Vertex AI and BigQuery, you can accelerate application deployment to stay at the forefront of AI innovation.
-
10
RedisJSON
RedisJSON - a JSON data type for Redis
RedisJSON is a Redis module that implements ECMA-404 The JSON Data Interchange Standard as a native data type. It allows storing, updating and fetching JSON values from Redis keys (documents). -
11
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t run... -
12
Roach
The complete web scraping toolkit for PHP
Roach is a complete web scraping toolkit for PHP. It is a shameless clone heavily inspired by the popular Scrapy package for Python. Roach allows us to define spiders that crawl and scrape web documents. But wait, there’s more. Roach isn’t just a simple crawler, but includes an entire pipeline to clean, persist and otherwise process extracted data as well. It’s your all-in-one resource for web scraping in PHP. Roach doesn’t depend on a specific framework. Instead, you can use the core package... -
13
WalletConnect v2.x.x
WalletConnect Monorepo
... python cli is linked (required to build some npm modules). Install workspace dependencies i.e. run npm install from root folder. Install redis. We recommend running it as a brew service. Pull and start ts-relay server (separate repo) Ensure everything runs correctly by executing npm run check. -
14
Log Viewer
Fast and beautiful Log Viewer for Laravel
Beautiful, fast, and open-source viewer for your app logs. Forget scrolling through raw log files to find the errors or debug logs you're looking for. Forget about SSH'ing onto the production server just to read the logs. Log Viewer helps you quickly and clearly see individual log entries, to search, filter, and make sense of your Laravel logs fast. It is free and easy to install. -
15
ProxyPool
An Efficient ProxyPool with Getter, Tester and Server
Simple and efficient proxy pool, providing the following functions. Regularly crawl free proxy websites, easy and scalable. Use Redis to store brokers and sort broker availability. Regular testing and screening to eliminate unavailable proxies and leave available proxies. Provides a proxy API to randomly select available proxies that pass the test. The principle analysis of the proxy pool can be seen in " How to Build an Efficient Proxy Pool ". It is recommended to read it before using... -
16
Scrapyd
A service daemon to run Scrapy spiders
Scrapyd can manage multiple projects and each project can have multiple versions uploaded, but only the latest one will be used for launching new spiders. A common (and useful) convention to use for the version name is the revision number of the version control tool you’re using to track your Scrapy project code. For example: r23. The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd... -
17
Acl
A powerful server and network library, including coroutine
...//MQTT/Redis/Memcached/Beanstalk/Handler Socket are implemented in Acl, and the codec library such as XML/JSON/MIME/BASE64/UUCODE/QPCODE/RFC2047/RFC1035, etc., are also included in Acl. Acl also provides a unified abstract interface for popular databases such as Mysql, Postgresql, Sqlite. Using Acl library users can write database applications more easily, quickly and safely. -
18
Bull
Queue package for handling distributed jobs and messages in NodeJS
The fastest, most reliable, Redis-based queue for Node. Carefully written for rock solid stability and atomicity. If you want to start using the next major version of Bull written entirely in Typescript you are welcome to the new repo here. Otherwise you are very welcome to still use Bull, which is a safe, battle tested codebase. If you need high quality production Redis instances for your Bull projects, please consider subscribing to RedisGreen, leaders in Redis hosting that works perfectly... -
19
Carmine
Redis client and message queue for Clojure
A pure-Clojure Redis client & message queue. Redis is awesome and it's getting more awesome every day. It deserves a great Clojure client. Fully documented, API with full support for the latest Redis versions. Industrial strength connection pooling. Composable, first-class command functions. Flexible, high-performance binary-safe serialization using Nippy. Simple, high-performance message queue (v2+, Redis 2.6+). Simple, high-performance distributed lock (v2+, Redis 2.6+). Pluggable compression... -
20
NautilusTrader
A high-performance algorithmic trading platform
NautilusTrader is an open-source, high-performance, production-grade algorithmic trading platform, provides quantitative traders with the ability to backtest portfolios of automated trading strategies on historical data with an event-driven engine, and also deploy those same strategies live, with no code changes. The platform is 'AI-first', designed to develop and deploy algorithmic trading strategies within a highly performant and robust Python native environment. This helps to address the... -
21
Ccache
A fast compiler cache
.... using Nginx or Google Cloud Storage), Redis or local filesystem, optionally sharding data onto a server cluster. Supports fast "direct" and "depend" modes that don't rely on using the preprocessor. Supports compression using Zstandard. Checksums cache content using XXH3 to detect data corruption. Keeps statistics on hits/misses. Automatic cache size management. Easy installation. -
22
Jedis
A small redis java client
Jedis is a blazingly small and sane Redis java client. Jedis was conceived to be EASY to use. Jedis is fully compatible with redis 2.8.x, 3.x.x and above. The features supported are sorting, connection handling, commands operating on any kind of values, commands operating on string values, commands operating on hashes, commands operating on lists, commands operating on sets, commands operating on sorted sets, commands operating on streams, transactions, pipelining, publish/subscribe... -
23
jackal
Instant messaging server for the Extensible Messaging
jackal is a free, open-source, high-performance XMPP server that aims to be known for its stability, simple configuration and low resource consumption. Customizable, enforced SSL/TLS. Stream compression (zlib), database connectivity for storing offline messages and user settings (PostgreSQL 9.5+, BoltDB). Caching (Redis 6.2+), clustering capabilities (etcd 3.4+) Expose Prometheus metrics, cross-platform (OS X, Linux). To make it easy to install jackal via Helm in Kubernetes a chart has been... -
24
Redigo
Go client for Redis
Redigo is a Go client for the Redis database. A Print-like API with support for all Redis commands. Pipelining, including pipelined transactions. Publish/Subscribe. Connection pooling. Script helper type with optimistic use of EVALSHA. Helper functions for working with command replies. The Conn interface is the primary interface for working with Redis. Applications create connections by calling the Dial, DialWithTimeout or NewConn functions. In the future, functions will be added for creating... -
25
Lettuce
Advanced Java Redis client for thread-safe sync, async, etc.
Build elastic data access. Lettuce is a scalable Redis client for building non-blocking Reactive applications. Lettuce is a fully non-blocking Redis client built with netty providing Reactive, Asynchronous and Synchronous Data Access. Lettuce provides asynchronous API with RedisFuture (CompletionStage) and Reactive types Flux [N] and Mono [0|1]. Low-latency communication, backpressure-enabled network engine for NIO TCP, epoll TCP and Unix Domain Sockets. Reactive Streaming is fully supported...