Search Results for "scrapy-redis"
Sort By:
Showing 214 open source projects for "scrapy-redis"
View related business solutions-
Give your HR team the tools they need to streamline administrative tasks, support employees, and make informed decisions with the OrangeHRM free and open source HR software.
-
Whether natural disaster, cyberattack, or plain-old human error, data can disappear in the blink of an eye. ConnectWise BCDR (formerly Recover) delivers reliable and secure backup and disaster recovery backed by powerful automation and a 24/7 NOC to get your clients back to work in minutes, not days.
-
1
Scrapy-Redis
Redis-based components for Scrapy
You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3... -
2
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
3
Redis
An in-memory database that persists on disk
Redis is an open-source (BSD licensed), in-memory data structure store, used as a database, cache, and message broker. Redis provides data structures such as strings, hashes, lists, sets, sorted sets with range queries, bitmaps, hyperloglogs, geospatial indexes, and streams. Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. You... -
4
Redis for Windows
Native port of Redis for Windows
Native port of Redis for Windows. Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported, Strings, Lists, Sets, Sorted Sets, Hashes, Streams, HyperLogLogs. This repository contains an unofficial port of Redis to Windows. At the moment win-4.0.14 branch provides a stable port of Redis 4.0.14 for Windows x64 and win-5.0 branch provides a stable port of Redis 5.0.14 for Windows x64, both merged with archived port of win... -
ConnectWise CPQ, formerly ConnectWise Sell, is a professional quote and proposal automation software for IT solution providers. ConnectWise CPQ offers a wide range of tools that enables IT solution providers to save time, quote more, and win big. Top features include professional quote or proposal templates, product catalog and sourcing, workflow automation, sales reporting, and integrations with best-in-breed solutions like Cisco, Dell, HP, and Salesforce.
-
5
Redis Store
Cache Redis stores for Ruby web frameworks
Redis Store provides a full set of stores (Cache, I18n, Session, HTTP Cache) for modern Ruby frameworks like: Ruby on Rails, Sinatra, Rack, Rack::Cache and I18n. It supports object marshaling, timeouts, single or multiple nodes, and namespaces. -
6
redis-operator
Redis Operator creates/configures/manages high availability redis
Redis Operator creates/configures/manages redis-failovers atop Kubernetes. Kubernetes version: 1.21 or higher Redis version: 6 or higher. Redis operator is being tested against Kubernetes 1.25 1.26 1.27 and redis 6 All dependencies have been vendored, so there's no need to any additional downloads. -
7
Node Redis
A high performance Node.js Redis client
This library is a 1 to 1 mapping of the Redis commands. Each Redis command is exposed as a function on the client object. All functions take either an args Array plus optional callback Function or a variable number of individual arguments followed by an optional callback. Node Redis currently doesn't natively support promises (this is coming in v4), however you can wrap the methods you want to use with promises using the built-in Node.js util.promisify method on Node.js >= v8. Care should... -
8
graphql-redis-subscriptions
A graphql subscriptions implementation using redis
This package implements the PubSubEngine Interface from the graphql-subscriptions package and also the new AsyncIterator interface. It allows you to connect your subscriptions manager to a Redis Pub Sub mechanism to support multiple subscription manager instances. As the graphql-subscriptions package is declared as a peer dependency, you might receive a warning about an unmet peer dependency if it's not installed already by one of your other packages. Calling the method asyncIterator... -
9
Prometheus Redis Metrics Exporter
Prometheus Exporter for Redis Metrics. Supports Redis 2.x, 3.x, 4.x, 5
Prometheus exporter for Redis metrics. Supports Redis 2.x, 3.x, 4.x, 5.x, 6.x, and 7.x. To have instances in the drop-down as human readable names rather than IPs, it is suggested to use instance relabelling. The Prometheus docs have a very informative article on how multi-target exporters are intended to work. Run the exporter with the command line flag --redis.addr= so it won't try to access the local instance every time the /metrics endpoint is scraped. If authentication is needed... -
Exact Payments delivers easy-to-integrate embedded payment solutions enabling you to rapidly onboard merchants, instantly activate a variety of payment methods and accelerate your revenue — delivering an end-to-end payment processing platform for SaaS businesses.
-
10
Spring Data Redis
Provides support to increase developer productivity in Java
Provides support to increase developer productivity in Java when using Redis, a key-value store. Uses familiar Spring concepts such as a template class for core API usage and lightweight repository-style data access. The primary goal of the Spring Data project is to make it easier to build Spring-powered applications that use new data access technologies such as non-relational databases, map-reduce frameworks, and cloud-based data services. Connection package as low-level abstraction across... -
11
DbGate
Database manager for MySQL, PostgreSQL, SQL Server, MongoDB
Database manager for MySQL, PostgreSQL, SQL Server, MongoDB, SQLite and others. Runs under Windows, Linux, Mac or as web application. DbGate is a cross-platform database manager. It's designed to be simple to use and effective when working with more databases simultaneously. But there are also many advanced features like schema compare, visual query designer, chart visualization or batch export and import. Mongo JavaScript editor, execute Mongo script (with NodeJs syntax). Redis tree view... -
12
RDM
Cross-platform GUI management tool for Redis
Cross-platform open source database management tool for Redis. RDM is easy-to-use GUI tool available for Windows, Linux, MacOS and iPadOS. Analyse your Redis ® memory usage with RDM and remove obsolete data with bulk removal. RDM supports all latest Redis ® features: ACL, Streams, Cluster, Sentinel, ReJSON module, HyperLogLog, etc. Build-in TLS, SSH and TLS-over-SSH tunneling for easy and secure access to any redis-server. RDM works with Amazon ElastiCache, Microsoft Azure Redis Cache, Digital... -
13
ProxyPool
An Efficient ProxyPool with Getter, Tester and Server
Simple and efficient proxy pool, providing the following functions. Regularly crawl free proxy websites, easy and scalable. Use Redis to store brokers and sort broker availability. Regular testing and screening to eliminate unavailable proxies and leave available proxies. Provides a proxy API to randomly select available proxies that pass the test. The principle analysis of the proxy pool can be seen in " How to Build an Efficient Proxy Pool ". It is recommended to read it before using... -
14
RedisJSON
RedisJSON - a JSON data type for Redis
RedisJSON is a Redis module that implements ECMA-404 The JSON Data Interchange Standard as a native data type. It allows storing, updating and fetching JSON values from Redis keys (documents). -
15
Bull
Queue package for handling distributed jobs and messages in NodeJS
The fastest, most reliable, Redis-based queue for Node. Carefully written for rock solid stability and atomicity. If you want to start using the next major version of Bull written entirely in Typescript you are welcome to the new repo here. Otherwise you are very welcome to still use Bull, which is a safe, battle tested codebase. If you need high quality production Redis instances for your Bull projects, please consider subscribing to RedisGreen, leaders in Redis hosting that works perfectly... -
16
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t run... -
17
Ccache
A fast compiler cache
.... using Nginx or Google Cloud Storage), Redis or local filesystem, optionally sharding data onto a server cluster. Supports fast "direct" and "depend" modes that don't rely on using the preprocessor. Supports compression using Zstandard. Checksums cache content using XXH3 to detect data corruption. Keeps statistics on hits/misses. Automatic cache size management. Easy installation. -
18
Jedis
A small redis java client
Jedis is a blazingly small and sane Redis java client. Jedis was conceived to be EASY to use. Jedis is fully compatible with redis 2.8.x, 3.x.x and above. The features supported are sorting, connection handling, commands operating on any kind of values, commands operating on string values, commands operating on hashes, commands operating on lists, commands operating on sets, commands operating on sorted sets, commands operating on streams, transactions, pipelining, publish/subscribe... -
19
ioredis
A performance-focused Redis client for Node.js
ioredis is a robust, full-featured Redis client that is used in the world's biggest online commerce company Alibaba and many other awesome companies. Full-featured. It supports Cluster, Sentinel, Streams, Pipelining and of course Lua scripting & Pub/Sub (with the support of binary messages). High performance. Delightful API. It works with Node callbacks and Native promises. Transformation of command arguments and replies. Transparent key prefixing. Abstraction for Lua scripting, allowing you... -
20
POCO
Cross-platform C++ libraries for building network applications
The POCO C++ Libraries are powerful cross-platform C++ libraries for building network- and internet-based applications that run on desktop, server, mobile, IoT, and embedded systems. Whether building automation systems, industrial automation, IoT platforms, air traffic management systems, enterprise IT application and infrastructure management, security and network analytics, automotive infotainment and telematics, financial or healthcare, C++ developers have been trusting the POCO C++... -
21
Acl
A powerful server and network library, including coroutine
...//MQTT/Redis/Memcached/Beanstalk/Handler Socket are implemented in Acl, and the codec library such as XML/JSON/MIME/BASE64/UUCODE/QPCODE/RFC2047/RFC1035, etc., are also included in Acl. Acl also provides a unified abstract interface for popular databases such as Mysql, Postgresql, Sqlite. Using Acl library users can write database applications more easily, quickly and safely. -
22
DDEV
Docker-based local PHP+Node.js web development environments
Container superpowers with zero required Docker skills: environments in minutes, multiple concurrent projects, and less time to deployment. Fast spinup and easy sharing let everyone skip the devops and get back to the code. Use the exact same setup across machines, team members, and even hosted environments. Focus on the software you need for each project without installing anything on your workstation. Agencies, enterprise businesses, and open source communities have trusted and shaped... -
23
Centrifugo
Scalable real-time messaging server in a language-agnostic way
Centrifugo is language-agnostic. It's a standalone server with a simple API that integrates well with an application written in any programming language. No need to change an existing application architecture to introduce real-time features. Just add Centrifugo nearby and let it deal with persistent connections. Centrifugo is built in Go language with some smart optimizations inside. It has good performance – see a description of a test stand with one million WebSocket connections and 30... -
24
Scrapyd
A service daemon to run Scrapy spiders
Scrapyd can manage multiple projects and each project can have multiple versions uploaded, but only the latest one will be used for launching new spiders. A common (and useful) convention to use for the version name is the revision number of the version control tool you’re using to track your Scrapy project code. For example: r23. The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd... -
25
Kvrocks
NoSQL database that uses RocksDB as storage engine
Apache Kvrocks(Incubating) is a distributed key-value NoSQL database that uses RocksDB as storage engine and is compatible with Redis protocol. Kvrocks intends to decrease the cost of memory and increase the capacity while compared to Redis. The design of replication and storage was inspired by rocksplicator and blackwidow. Redis protocol, user can use redis client to visit the kvrocks. Namespace, similar to redis db but use token per namespace. Replication, async replication using binlog like...