Search Results for "queue"

You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions.

pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. Since pyspider has various components, you can just run pyspider to start a standalone and third service free instance. ...

This is a "fork" of the ImageFap Gallery Downloader project to change it from querying keyboard and instead allow you to pass the URLs along via CLI. Additional features will be added as requested or desired.

Note: the work on this project is currently on halt. Check out https://sourceforge.net/projects/imagefap-gallery-dl-cli/?source=directory for a more recent version. ImageFap Gallery Downloader is a Python script for full image gallery downloads on ImageFap and xHamster. xHamster support added in v0.3 In order to download user folders in v0.4, open the user profile, follow the 'Galleries' link and copy the folder link from the sidebar to the clipboard. Download everything from a user...

Sequentially downloads anime episodes from www.realitylapse.com with FREE DOWNLOAD MANAGER

A Console-based BitTorrent Client with built-in scheduler for handling multiple sessions. It is designed to manage sessions in queue easily without heavy-weight GUI. External module can search for new torrents in trackers and submit it automatically.