Search Results for "redis for windows"

You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version 0.3...

An essential toolset that eases WordPress site and server administration.

... SQLite, Redis, MongoDB, and DynamoDB; or save responses as plain JSON files, YAML, and more. Use Cache-Control and other standard HTTP headers, define your own expiration schedule, and keep your cache clutter-free with backends that natively support TTL or any combination of strategies. Works out of the box with zero config, but with a robust set of features for configuring and extending the library to suit your needs.

pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. Since pyspider has various components, you can just run pyspider to start a standalone and...

resn is a library to create social networking apps easily using Redis and Python. Friends, followers, feed and authentication features provided out of the box, works with all web frameworks.