Search Results for "scrapy-redis"
Sort By:
Showing 8 open source projects for "scrapy-redis"
View related business solutions-
Payroll Services for Small Businesses | QuickBooksEasily pay your team and access powerful tools, employee benefits, and supportive experts with the #1 online payroll service provider. Manage payroll and access HR and employee services in one place. Pay your team automatically once your payroll setup is complete. We'll calculate, file, and pay your payroll taxes automatically.
-
Never Get Blocked Again | Enterprise Web ScrapingGet unrestricted access to public web data with our ethically-sourced proxy network. Automated session management and advanced unblocking handle the hard parts. Scale from 1 to 1M requests with zero blocks. Built for developers with ready-to-use APIs, serverless functions, and complete documentation. Used by 20,000+ companies including Fortune 500s. SOC2 and GDPR compliant.
-
1
Gerapy
Distributed Crawler Management Framework Based on Scrapy
Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Someone who has worked as a crawler with Python may use Scrapy. Scrapy is indeed a very powerful crawler framework. It has high crawling efficiency and good scalability. It is basically a necessary tool for developing crawlers using Python. If you use Scrapy as a crawler, then of course we can use our own host to crawl when crawling, but when the crawl is very large, we can’t run... -
2
ProxyPool
An Efficient ProxyPool with Getter, Tester and Server
Simple and efficient proxy pool, providing the following functions. Regularly crawl free proxy websites, easy and scalable. Use Redis to store brokers and sort broker availability. Regular testing and screening to eliminate unavailable proxies and leave available proxies. Provides a proxy API to randomly select available proxies that pass the test. The principle analysis of the proxy pool can be seen in " How to Build an Efficient Proxy Pool ". It is recommended to read it before using... -
3
Django RQ
A simple app that provides django integration for RQ
A simple app that provides django integration for RQ (Redis Queue). Django integration with RQ, a Redis-based Python queuing library. Django-RQ is a simple app that allows you to configure your queues in django's settings.py and easily use them in your project. Django-RQ allows you to easily put jobs into any of the queues defined in settings.py. You can provide your own singleton Redis connection object to this function so that it will not create a new connection object for each queue... -
4
dynaconf
Configuration Management for Python
... Vault and Redis as settings and secrets storage. Built-in extensions for Django and Flask web frameworks. CLI for common operations such as init, list, write, validate, export. On your own code you import and use settings object imported from your config.py file. Dynaconf prioritizes the use of environment variables and you can optionally store settings in Settings Files using any of toml|yaml|json|ini|py extension. -
The #1 Embedded Analytics Solution for SaaS Teams.Qrvey’s comprehensive embedded analytics software enables you to design more customizable analytics experiences for your end users.
-
5
OpsManage
Automated operation and maintenance platform
..., Python3.6 + HTML + JScripts. Front-end web framework, Bootstrap. Backend Web Framework, Django. Backend Task Framework, Celery + Redis. Programming language is Python 3.6. Operating System is CentOS 6+. Ansible version 2.6+. Deployment platform and node server Rsync 3+. -
6
Empact Foundation Class Library
Cross-platform C++ library for use as a default application framework.
A mature cross-platform C++ library for use as a default application framework. Features include: * Threading & synchronization * Socket programming: SSL, NanoMsg & ZMQ * File I/O utilities: zlib, ini, yaml * Native Database access: MySQL, SQLite, BerkleyDB, Postgre, REDIS and ODBC * Built-in mini XML parser; optional EXPAT, LIBXML and MSXML support * Network protocol stack: HTTP, FTP, SMTP, POP3, SOAP, XMLRPC * Scripting languages: Perl, Python, JavaScript, VBScript, Java, Lua, TCL... -
7
django-dynamic-scraper
Creating Scrapy scrapers via the Django admin interface
Django Dynamic Scraper (DDS) is an app for Django build on top of the scraping framework Scrapy. While preserving many of the features of Scrapy it lets you dynamically create and manage spiders via the Django admin interface. With Django Dynamic Scraper (DDS) you can define your Scrapy scrapers dynamically via the Django admin interface and save your scraped items in the database you defined for your Django project. Since it simplifies things DDS is not usable for all kinds of scrapers... -
8
PROTON
High-level python framework that facilitates rapid server-side develop
PROTON is a high-level Python framework that facilitates rapid server-side development with clean & pragmatic design. Thanks for checking it out! PROTON aims at easing server-side development for all Python enthusiasts. Essentially, by running a shell command, developer will auto generate necessary Model, Controller and APIs! All of this with connectivity to Transactional Databases (PROTON supports Postgresql, MySQL & SQL Server),caching (Redis middleware), Auto generated OpenAPI specs...
- Previous
- You're on page 1
- Next