Search Results for "open queue"

You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions. Note that pickle is not compatible between python versions. Version...

Crawlee is a web scraping and browser automation library. It helps you build reliable crawlers. Fast. Crawlee won't fix broken selectors for you (yet), but it helps you build and maintain your crawlers faster. When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back. It keeps your proxies healthy by rotating them smartly with good fingerprints that...

OmniPull is a powerful, cross-platform download manager built with Python and PySide6. It provides a modern, intuitive interface for managing downloads with advanced features like multi-threading, queue management, and media extraction.

DownThemAll is a powerful yet easy-to-use extension that adds new advanced download capabilities to your browser. DownThemAll lets you download all the links or images on a website and much more: you can refine your downloads by fully customizable filters to get only what you really want. DownThemAll is all you can desire from a download manager: it allows you to queue, pause and resume downloads at any time and is fully integrated in your favorite browser! This is the WebExtension version...

A jQuery plugin for file uploading using ajax(a sync); includes support for queues, progress tracking and drag and drop. Very configurable and easy to adapt to any Frontend design, and very easy to work along side any backend logic. The focus will be modern browsers, but also providing a method to know when the plugin is not supported. The idea is to keep it simple and lightweight. Basic Javascript knowledge is necessary to setup this plugin: how to set settings, callback events, etc....