Showing 4 open source projects for "queue"

View related business solutions
  • Our Free Plans just got better! | Auth0 Icon
    Our Free Plans just got better! | Auth0

    With up to 25k MAUs and unlimited Okta connections, our Free Plan lets you focus on what you do best—building great apps.

    You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
    Try free now
  • Workload Automation for Global Enterprises Icon
    Workload Automation for Global Enterprises

    Orchestrate Your Entire Tech Stack with Redwood RunMyJobs

    Redwood lets you orchestrate securely and reliably across any application, service or server, in the cloud or on-premise, all inside a single platform.
    Learn More
  • 1
    Scrapy-Redis

    Scrapy-Redis

    Redis-based components for Scrapy

    You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls. Scraped items gets pushed into a redis queued meaning that you can start as many as needed post-processing processes sharing the items queue. Scheduler + Duplication Filter, Item Pipeline, Base Spiders. Default requests serializer is pickle, but it can be changed to any module with loads and dumps functions.
    Downloads: 0 This Week
    Last Update:
    See Project
  • 2
    crawlee

    crawlee

    A web scraping and browser automation library for Node.js

    Crawlee is a web scraping and browser automation library. It helps you build reliable crawlers. Fast. Crawlee won't fix broken selectors for you (yet), but it helps you build and maintain your crawlers faster. When a website adds JavaScript rendering, you don't have to rewrite everything, only switch to one of the browser crawlers. When you later find a great API to speed up your crawls, flip the switch back. It keeps your proxies healthy by rotating them smartly with good fingerprints that...
    Downloads: 1 This Week
    Last Update:
    See Project
  • 3

    webotron

    Using industrial automation techniques for creating web scraping tools

    Industry uses machines that can easily maim or kill their operators and is also used in very adverse environments. In spite of this, production quality must be close to perfect without reliance on operator skill or attentiveness. Control programs must be robust, yet simple enough to be understood and maintained by non programmer skilled trades like electricians . The main programming model is the PLC which implements double buffering and an event loop. The most advanced production model...
    Downloads: 0 This Week
    Last Update:
    See Project
  • 4
    pyspider

    pyspider

    A powerful Spider(Web Crawler) system in Python

    pyspider is a powerful Spider(Web Crawler) system in Python. Components are connected by message queue. Every component, including message queue, is running in their own process/thread, and replaceable. That means, when process is slow, you can have many instances of processor and make full use of multiple CPUs, or deploy to multiple machines. This architecture makes pyspider really fast. benchmarking. Since pyspider has various components, you can just run pyspider to start a standalone and third service free instance. ...
    Downloads: 0 This Week
    Last Update:
    See Project
  • File Synchronization, File Replication and File Archiving software solutions. Icon
    File Synchronization, File Replication and File Archiving software solutions.

    SIMPLIFY CRITICAL FILE TRANSFERS

    SureSync is a file replication and synchronization application that provides one-way and multi-way processing in both scheduled and real-time modes.
    Learn More
  • Previous
  • You're on page 1
  • Next