Search Results for "scrapy-redis"
Sort By:
Showing 12 open source projects for "scrapy-redis"
View related business solutions-
From improving customer experience through seamless sign-on to making MFA as easy as a click of a button – your login box must find the right balance between user convenience, privacy and security.
-
Red Hat Enterprise Linux (RHEL) on Microsoft Azure provides a secure, reliable, and flexible foundation for your cloud infrastructure. Red Hat Enterprise Linux on Microsoft Azure is ideal for enterprises seeking to enhance their cloud environment with seamless integration, consistent performance, and comprehensive support.
-
1
Scrapy
A fast, high-level web crawling and web scraping framework
Scrapy is a fast, open source, high-level framework for crawling websites and extracting structured data from these websites. Portable and written in Python, it can run on Windows, Linux, macOS and BSD. Scrapy is powerful, fast and simple, and also easily extensible. Simply write the rules to extract the data, and add new functionality if you wish without having to touch the core. Scrapy does the rest, and can be used in a number of applications. It can be used for data mining, monitoring... -
2
ProxyPool
An Efficient ProxyPool with Getter, Tester and Server
Simple and efficient proxy pool, providing the following functions. Regularly crawl free proxy websites, easy and scalable. Use Redis to store brokers and sort broker availability. Regular testing and screening to eliminate unavailable proxies and leave available proxies. Provides a proxy API to randomly select available proxies that pass the test. The principle analysis of the proxy pool can be seen in " How to Build an Efficient Proxy Pool ". It is recommended to read it before using... -
3
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t run... -
4
Django RQ
A simple app that provides django integration for RQ
A simple app that provides django integration for RQ (Redis Queue). Django integration with RQ, a Redis-based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in django's settings.py and easily use them in your project. Django-RQ allows you to easily put jobs into any of the queues defined in settings.py. You can provide your own singleton Redis connection object to this function so that it will not create a new connection object for each queue... -
Built for growing businesses and well-established enterprises alike, Multiview is a highly scalable and robust ERP.
-
5
dynaconf
Configuration Management for Python
... Vault and Redis as settings and secrets storage. Built-in extensions for Django and Flask web frameworks. CLI for common operations such as init, list, write, validate, export. On your own code you import and use settings object imported from your config.py file. Dynaconf prioritizes the use of environment variables and you can optionally store settings in Settings Files using any of toml|yaml|json|ini|py extension. -
6
OpsManage
Automated operation and maintenance platform
..., Python3.6 + HTML + JScripts. Front-end web framework, Bootstrap. Backend Web Framework, Django. Backend Task Framework, Celery + Redis. Programming language is Python 3.6. Operating System is CentOS 6+. Ansible version 2.6+. Deployment platform and node server Rsync 3+. -
7
Pottery
Redis for humans
Redis is awesome, but Redis commands are not always intuitive. Pottery is a Pythonic way to access Redis. If you know how to use Python dicts, then you already know how to use Pottery. Pottery is useful for accessing Redis more easily, and also for implementing microservice resilience patterns, and it has been battle-tested in production at scale. -
8
Empact Foundation Class Library
Cross-platform C++ library for use as a default application framework.
A mature cross-platform C++ library for use as a default application framework. Features include: * Threading & synchronization * Socket programming: SSL, NanoMsg & ZMQ * File I/O utilities: zlib, ini, yaml * Native Database access: MySQL, SQLite, BerkleyDB, Postgre, REDIS and ODBC * Built-in mini XML parser; optional EXPAT, LIBXML and MSXML support * Network protocol stack: HTTP, FTP, SMTP, POP3, SOAP, XMLRPC * Scripting languages: Perl, Python, JavaScript, VBScript, Java, Lua, TCL, Squirrel... -
9
django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
Django Dynamic Scraper (DDS) is an app for Django build on top of the scraping framework Scrapy. While preserving many of the features of Scrapy it lets you dynamically create and manage spiders via the Django admin interface. With Django Dynamic Scraper (DDS) you can define your Scrapy scrapers dynamically via the Django admin interface and save your scraped items in the database you defined for your Django project. Since it simplifies things DDS is not usable for all kinds of scrapers... -
ConnectWise ScreenConnect, formerly ConnectWise Control, is a remote support solution for Managed Service Providers (MSP), Value Added Resellers (VAR), internal IT teams, and managed security providers. Fast, reliable, secure, and simple to use, ConnectWise ScreenConnect helps businesses solve their customers' issues faster from any location. The platform features remote support, remote access, remote meeting, customization, and integrations with leading business tools.
-
10
RedMQ
Simple, flexible messaging with Redis.
RedMQ is a Redis-based messaging system that is specifically designed to support flexible messaging schemes, exposing an API that allows messages to be filtered by a number of message attributes. At its core is a Python TCP server built on Asynchronous Messaging Protocol and Twisted's AMP module. This means RedMQ can support clients in any language, though only two clients - PHP and Python - have been implemented to date. The API is designed with flexibility in mind, providing native support... -
11
OPEW
A complete multi-language developement stack for GNU/Linux
... DBMS (MySQL, PostgreSQL, MongoDB, SQLite3 and Redis). Git (SCM) is included as well with some tools for code versioning and review. OPEW is fully binary independent compiled stack and fully portable In the future OPEW aims to be extensible using addons then can be installed in the same stack improving itself. Currently OPEW is still an experimental beta version. -
12
PROTON
High-level python framework that facilitates rapid server-side develop
PROTON is a high-level Python framework that facilitates rapid server-side development with clean & pragmatic design. Thanks for checking it out! PROTON aims at easing server-side development for all Python enthusiasts. Essentially, by running a shell command, developer will auto generate necessary Model, Controller and APIs! All of this with connectivity to Transactional Databases (PROTON supports Postgresql, MySQL & SQL Server),caching (Redis middleware), Auto generated OpenAPI specs...
- Previous
- You're on page 1
- Next