Search Results for "scrapy-redis"
Sort By:
Showing 27 open source projects for "scrapy-redis"
View related business solutions-
Deploy Red Hat Ansible Automation Platform on Microsoft Azure for a strategic automation solution that allows you to orchestrate, govern and operationalize your Azure environment.
-
From improving customer experience through seamless sign-on to making MFA as easy as a click of a button – your login box must find the right balance between user convenience, privacy and security.
-
1
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
2
Scrapy-Redis
Redis-based components for Scrapy
You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3... -
3
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t run... -
4
Acl
A powerful server and network library, including coroutine
...//MQTT/Redis/Memcached/Beanstalk/Handler Socket are implemented in Acl, and the codec library such as XML/JSON/MIME/BASE64/UUCODE/QPCODE/RFC2047/RFC1035, etc., are also included in Acl. Acl also provides a unified abstract interface for popular databases such as Mysql, Postgresql, Sqlite. Using Acl library users can write database applications more easily, quickly and safely. -
BrightGauge, a ConnectWise solution, was started in 2011 to fill a missing need in the small-to-medium IT Services industry: a better way to manage data and provide the value of work to clients. BrightGauge Software allows you to display all of your important business metrics in one place through the use of gauges, dashboards, and client reports. Used by more than 1,800 companies worldwide, BrightGauge integrates with popular business solutions on the market, like ConnectWise, Continuum, Webroot, QuickBooks, Datto, IT Glue, Zendesk, Harvest, Smileback, and so many more. Dig deeper into your data by adding, subtracting, multiplying, and dividing one metric against another. BrightGauge automatically computes these formulas for you. Want to show your prospects how quick you are to respond to tickets? Show off your data with embeddable gauges on public sites.
-
5
Scrapyd
A service daemon to run Scrapy spiders
Scrapyd can manage multiple projects and each project can have multiple versions uploaded, but only the latest one will be used for launching new spiders. A common (and useful) convention to use for the version name is the revision number of the version control tool you’re using to track your Scrapy project code. For example: r23. The versions are not compared alphabetically but using a smarter algorithm (the same packaging uses) so r10 compares greater to r9, for example. Scrapyd... -
6
OAuth2 Proxy
A reverse proxy that provides authentication with Google, Azure, etc.
A reverse proxy and static file server that provides authentication using Providers (Google, GitHub, and others) to validate accounts by email, domain or group. If you are running a version older than v6.0.0 we strongly recommend you please update to the current version. After returning from the authentication provider, the OAuth tokens are stored in the configured session store (cookie, redis, ...) and a cookie is set. The request is forwarded to the upstream server with added user info... -
7
NodeBB
Node.js based forum software built for the modern web
A better community platform for the modern web. NodeBB is next generation forum software. It's powerful, mobile-ready and easy to use. NodeBB Forum Software is powered by Node.js and supports either Redis, MongoDB, or a PostgreSQL database. It utilizes web sockets for instant interactions and real-time notifications. NodeBB has many modern features out of the box such as social network integration and streaming discussions, while still making sure to be compatible with older browsers. From... -
8
Tyk API Gateway
Open Source API Gateway written in Go
... in the clouds as well as exposing APIs to teams, partners & consumers. Built from the ground up to be the fastest API gateway on the planet. It does not depend on a legacy proxy underneath. It has no 3rd party dependencies aside from Redis for distributed rate-limiting and token storage. Tyk Gateway can also be deployed as part of a larger Full Lifecycle API Management platform Tyk Self-Managed which also includes Management Control Plane, Dashboard GUI and Developer Portal. -
9
Roach
The complete web scraping toolkit for PHP
Roach is a complete web scraping toolkit for PHP. It is a shameless clone heavily inspired by the popular Scrapy package for Python. Roach allows us to define spiders that crawl and scrape web documents. But wait, there’s more. Roach isn’t just a simple crawler, but includes an entire pipeline to clean, persist and otherwise process extracted data as well. It’s your all-in-one resource for web scraping in PHP. Roach doesn’t depend on a specific framework. Instead, you can use the core package... -
As your users rely more and more on Cloud and Internet-based technologies, reliable internet connectivity becomes more and more important to your business. With Bigleaf’s proven SD-WAN architecture, groundbreaking AI, and DDoS attack mitigation, you can finally deliver the reliable internet connectivity your business needs without the limitations of traditional networking platforms. Bigleaf’s Cloud Access Network and plug-and-play router allow for limitless control to and from anywhere your traffic needs to go. Bigleaf’s self-driving AI automatically identifies and adapts to any changing circuit conditions and traffic needs—addressing issues before they impact your users. Bigleaf puts you in the driver’s seat of every complaint and support call with full-path traffic and network performance data, delivered as actionable insights, reports, and alerts.
-
10
ScrapydWeb
Web app for Scrapyd cluster management
Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization. Make sure that Scrapyd has been installed and started on all of your hosts. Start ScrapydWeb via command scrapydweb. (a config file would be generated for customizing settings on the first startup.) Add your Scrapyd servers, both formats of string and tuple are supported, you can attach basic auth for accessing the Scrapyd server, as well as a string for grouping or labeling. You can select any number... -
11
graceful
Gracefully exit HTTP servers (Express/Koa/Fastify/etc), databases
Gracefully exit HTTP servers (Express/Koa/Fastify/etc), databases (Mongo/Mongoose), Redis clients, Bree job schedulers, and custom handlers. See the Express, Koa, Fastify, or Other code snippet examples and Instance Options. You can pass Instance Options to customize your graceful handler (e.g. if you have more than one server, or wish to close both a Redis connection and a server at the same time). -
12
BeetleX
high performance dotnet core socket tcp communication components
High-performance dotnet core socket tcp communication components, support TLS, HTTP, HTTPS, WebSocket, RPC, Redis protocols, custom protocols and 1M connections problem solution. -
13
WordOps
Install and manage a high performance WordPress stack
An essential toolset that eases WordPress site and server administration. -
14
Routr
The future of programmable SIP servers
Lightweight sip proxy, location server, and registrar that provides a reliable and scalable SIP infrastructure for telephony carriers, communication service providers, and integrators. Routr provides all the tools required to deploy your VoIP network, including a command-line for remote server control. It can also be controlled via a RESTful API or a web-based GUI. Routr includes all tools for deploying your VoIP network. It offers remote server control via command-line, RESTful API, or a... -
15
Crawlab
Distributed web crawler admin platform for spiders management
Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium. Please use docker-compose to one-click to start up. By doing so, you don't even have to configure MongoDB database. The frontend app interacts with the master node, which communicates with other components such as MongoDB, SeaweedFS and worker nodes. Master node and worker nodes communicate... -
16
FastTunnel
Expose a local server to the internet
FastTunnel is a high-performance cross-platform intranet penetration tool. With it, you can expose intranet services to the public network for yourself or anyone to access. Unlike other penetration tools, the FastTunnel project is committed to creating an easy-to-extensible and easy-to-maintain intranet penetration framework. You can build your own penetration application by referencing the nuget package of FastTunnel.Core, and target the business extension functions you need. -
17
requests-cache
Persistent HTTP cache for python requests
... SQLite, Redis, MongoDB, and DynamoDB; or save responses as plain JSON files, YAML, and more. Use Cache-Control and other standard HTTP headers, define your own expiration schedule, and keep your cache clutter-free with backends that natively support TTL or any combination of strategies. Works out of the box with zero config, but with a robust set of features for configuring and extending the library to suit your needs. -
18
Sogou C++ Workflow
C++ parallel computing and asynchronous networking engine
As Sogou`s C++ server engine, Sogou C++ Workflow supports almost all back-end C++ online services of Sogou, including all search services, cloud input method, online advertisements, etc., handling more than 10 billion requests every day. This is an enterprise-level programming engine in light and elegant design which can satisfy most C++ back-end development requirements. -
19
FACIL.IO
Your high performance web application C framework
facil.io is an evented Network library written in C. facil.io provides high performance TCP/IP network services by using an evented design that was tested to provide an easy solution to the C10K problem. facil.io includes a mini-framework for Web Applications, with a fast HTTP / WebSocket server, integrated Pub/Sub, optional Redis connectivity, easy JSON handling, Mustache template rendering and more nifty tidbits. facil.io powers the HTTP/Websockets Ruby Iodine server and it can easily power... -
20
twemproxy
A fast, light-weight proxy for memcached and redis
twemproxy (pronounced "two-em-proxy"), aka nutcracker is a fast and lightweight proxy for memcached and redis protocol. It was built primarily to reduce the number of connections to the caching servers on the backend. This, together with protocol pipelining and sharding enables you to horizontally scale your distributed caching architecture. Fast and lightweight. Maintains persistent server connections. Keeps connection count on the backend caching servers low. Enables pipelining of requests... -
21
-
22
nuster
A high performance HTTP proxy cache server and RESTful NoSQL cache
... be used as an internal NoSQL cache that sits between your application and a database like Memcached or Redis as well as a user-facing NoSQL cache that sits between the end-user and your application. It supports headers, and cookies, so you can store per-user data to the same endpoint. -
23
N2O
Distributed WebSocket Application Server ISO 20922
N2O is an embeddable message protocol loop library for WebSocket, HTTP, MQTT and TCP servers. It provides basic features, such as process management, virtual nodes ring for request processing, sessions, frame encoding, and unified API for external mq and caching services. -
24
django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
Django Dynamic Scraper (DDS) is an app for Django build on top of the scraping framework Scrapy. While preserving many of the features of Scrapy it lets you dynamically create and manage spiders via the Django admin interface. With Django Dynamic Scraper (DDS) you can define your Scrapy scrapers dynamically via the Django admin interface and save your scraped items in the database you defined for your Django project. Since it simplifies things DDS is not usable for all kinds of scrapers... -
25
pyspider
A powerful Spider(Web Crawler) system in Python
pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. Since pyspider has various components, you can just run pyspider to start a standalone and...