Open Source Stream Processing Tools
Sort By:
Stream Processing Tools
View 57 business solutionsBrowse free open source Stream Processing tools and projects below. Use the toggles on the left to filter open source Stream Processing tools by OS, license, language, programming language, and project status.
-
Your monitoring isn't a stack. It's a pile. Fix that.Replace Datadog, New Relic, and Sentry without adding three more dashboards.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Best-of Python
A ranked list of awesome Python open-source libraries
This curated list contains 390 awesome open-source projects with a total of 1.4M stars grouped into 28 categories. All projects are ranked by a project-quality score, which is calculated based on various metrics automatically collected from GitHub and different package managers. If you like to add or update projects, feel free to open an issue, submit a pull request, or directly edit the projects.yaml. Contributions are very welcome! Ranked list of awesome python libraries for web development. Correctly generate plurals, ordinals, indefinite articles; convert numbers. Libraries for loading, collecting, and extracting data from a variety of data sources and formats. Libraries for data batch- and stream-processing, workflow automation, job scheduling, and other data pipeline tasks. -
2
CocoIndex
ETL framework to index data for AI, such as RAG
CocoIndex is an open-source framework designed for building powerful, local-first semantic search systems. It lets users index and retrieve content based on meaning rather than keywords, making it ideal for modern AI-based search applications. CocoIndex leverages vector embeddings and integrates with various models and frameworks, including OpenAI and Hugging Face, to provide high-quality semantic understanding. It’s built for transparency, ease of use, and local control over your search data, distinguishing itself from closed, black-box systems. The tool is suitable for developers working on personal knowledge bases, AI search interfaces, or private LLM applications. -
3
Acl
A powerful server and network library, including coroutine
The Acl (Advanced C/C++ Library) project a is powerful multi-platform network communication library and service framework, supporting LINUX, WIN32, Solaris, FreeBSD, MacOS, AndroidOS, iOS. Many applications written by Acl run on these devices with Linux, Windows, iPhone and Android and serve billions of users. There are some important modules in Acl project, including network communcation, server framework, application protocols, multiple coders, etc. The common protocols such as HTTP/SMTP/ICMP//MQTT/Redis/Memcached/Beanstalk/Handler Socket are implemented in Acl, and the codec library such as XML/JSON/MIME/BASE64/UUCODE/QPCODE/RFC2047/RFC1035, etc., are also included in Acl. Acl also provides a unified abstract interface for popular databases such as Mysql, Postgresql, Sqlite. Using Acl library users can write database applications more easily, quickly and safely. -
4
Numaflow
Kubernetes-native platform to run massively parallel data/streaming
Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices. Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platform. -
Our Free Plans just got better! | Auth0You asked, we delivered! Auth0 is excited to expand our Free and Paid plans to include more options so you can focus on building, deploying, and scaling applications without having to worry about your security. Auth0 now, thank yourself later.
-
5
fluentbit
Fast and Lightweight Logs and Metrics processor for Linux, BSD, OSX
Fluent Bit is a super-fast, lightweight, and highly scalable logging and metrics processor and forwarder. It is the preferred choice for cloud and containerized environments. A robust, lightweight, and portable architecture for high throughput with low CPU and memory usage from any data source to any destination. Proven across distributed cloud and container environments. Highly available with I/O handlers to store data for disaster recovery. Granular management of data parsing and routing. Filtering and enrichment to optimize security and minimize cost. The lightweight, asynchronous design optimizes resource usage: CPU, memory, disk I/O, network. No more OOM errors! Integration with all your technology, cloud-native services, containers, streaming processors, and data backends. Fully event-driven design leverages the operating system API for performance and reliability. All operations to collect and deliver data are asynchronous. -
6
Amadeus
Harmonious distributed data analysis in Rust
Amadeus is a high-performance, distributed data processing framework written in Rust, designed to offer an ergonomic and safe alternative to tools like Apache Spark. It provides both streaming and batch capabilities, allowing users to work with real-time and historical data at scale. Thanks to Rust’s memory safety and zero-cost abstractions, Amadeus delivers performance gains while reducing the complexity and bugs common in large-scale data pipelines. It emphasizes developer productivity through a fluent, expressive API and makes it easier to build composable and reliable data transformation pipelines without sacrificing speed or safety. -
7
Benthos
Fancy stream processing made operationally mundane
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads. It comes with a powerful mapping language, is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary, docker image, or serverless function, making it cloud native as heck. Delivery guarantees can be a dodgy subject. Benthos processes and acknowledges messages using an in-process transaction model with no need for any disk persisted state, so when connecting to at-least-once sources and sinks it's able to guarantee at-least-once delivery even in the event of crashes, disk corruption, or other unexpected server faults. This behaviour is the default and free of caveats, which also makes deploying and scaling Benthos much simpler. -
8
Bytewax
Python Stream Processing
Bytewax is a Python framework that simplifies event and stream processing. Because Bytewax couples the stream and event processing capabilities of Flink, Spark, and Kafka Streams with the friendly and familiar interface of Python, you can re-use the Python libraries you already know and love. Connect data sources, run stateful transformations, and write to various downstream systems with built-in connectors or existing Python libraries. Bytewax is a Python framework and Rust distributed processing engine that uses a dataflow computational model to provide parallelizable stream processing and event processing capabilities similar to Flink, Spark, and Kafka Streams. You can use Bytewax for a variety of workloads from moving data à la Kafka Connect style all the way to advanced online machine learning workloads. Bytewax is not limited to streaming applications but excels anywhere that data can be distributed at the input and output. -
9
Riemann
A network event stream processing system, in Clojure
Riemann aggregates events from your servers and applications with a powerful stream processing language. Send an email for every exception in your app. Track the latency distribution of your web app. See the top processes on any host, by memory and CPU. Combine statistics from every Riak node in your cluster and forward to Graphite. Track user activity from second to second. Riemann streams are just functions which accept an event. Events are just structs with some common fields like :host and :service You can use dozens of built-in streams for filtering, altering, and combining events, or write your own. Since Riemann's configuration is a Clojure program, its syntax is concise, regular, and extendable. Configuration-as-code minimizes boilerplate and gives you the flexibility to adapt to complex situations. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
10
Watermill
Building event-driven applications the easy way in Go
Go library for building event-driven applications. Our goal was to create a tool that is easy to understand, even by junior developers. It doesn't matter if you want to do Event-driven architecture, CQRS, Event Sourcing or just stream MySQL Binlog to Kafka. Watermill was designed to process hundreds of thousands of messages per second. Every component is built in a way that allows you to configure it for your needs. You can also implement your own middleware for the router. Watermill is using proven technologies and has a strong unit and integration tests coverage for critical areas. Watermill is a Go library for working efficiently with message streams. It is intended for building event driven applications, enabling event sourcing, RPC over messages, sagas and basically whatever else comes to your mind. You can use conventional pub/sub implementations like Kafka or RabbitMQ, but also HTTP or MySQL binlog if that fits your use case. -
11
MXQuery is a low-footprint implementation of XQuery 1.0, XQuery Update 1.0, XQuery Fulltext 1.0 and XQuery Scripting 1.0 as well as a subset of XQuery 1.1 (windowing, try/catch). It provides extensions to do data stream processing/CEP and SOAP/REST
-
12
Akka
Build concurrent, distributed, and resilient message-driven apps
Build powerful reactive, concurrent, and distributed applications more easily. Akka is a toolkit for building highly concurrent, distributed, and resilient message-driven applications for Java and Scala. Actors and Streams let you build systems that scale up, using the resources of a server more efficiently, and out, using multiple servers. Building on the principles of The Reactive Manifesto Akka allows you to write systems that self-heal and stay responsive in the face of failures. Up to 50 million msg/sec on a single machine. Small memory footprint; ~2.5 million actors per GB of heap. Distributed systems without single points of failure. Load balancing and adaptive routing across nodes. Event Sourcing and CQRS with Cluster Sharding. Distributed Data for eventual consistency using CRDTs. Asynchronous non-blocking stream processing with backpressure. -
13
Arroyo
Distributed stream processing engine in Rust
Arroyo is a distributed stream processing engine written in Rust, designed to efficiently perform stateful computations on streams of data. Unlike traditional batch processing, streaming engines can operate on both bounded and unbounded sources, emitting results as soon as they are available. -
14
An experimental CEP (Complex Event Processing) engine. It implements the event stream processing as a library embeddable in C++ and Perl. Since then it has been renamed to Triceps, so please look at the new location https://sourceforge.net/projects/t
-
15
Cosmos DB Spark
Apache Spark Connector for Azure Cosmos DB
Azure Cosmos DB Spark is the official connector for Azure CosmosDB and Apache Spark. The connector allows you to easily read to and write from Azure Cosmos DB via Apache Spark DataFrames in Python and Scala. It also allows you to easily create a lambda architecture for batch-processing, stream-processing, and a serving layer while being globally replicated and minimizing the latency involved in working with big data. -
16
DSPatch
The Refreshingly Simple C++ Dataflow Framework
Webite: http://flowbasedprogramming.com DSPatch, pronounced "dispatch", is a powerful C++ dataflow framework. DSPatch is not limited to any particular domain or data type, from reactive programming to stream processing, DSPatch's generic, object-oriented API allows you to create virtually any dataflow system imaginable. *See also:* DSPatcher ( https://github.com/MarcusTomlinson/DSPatcher ): A cross-platform graphical tool for building DSPatch circuits. DSPatchables ( https://github.com/MarcusTomlinson/DSPatchables ): A DSPatch component repository. -
17
Dataflow Java SDK
Google Cloud Dataflow provides a simple, powerful model
The Dataflow Java SDK is the open-source Java library that powers Apache Beam pipelines for Google Cloud Dataflow, a serverless and scalable platform for processing large datasets in both batch and stream modes. This SDK allows developers to write Beam-based pipelines in Java and execute them on Dataflow, taking advantage of features like autoscaling, dynamic work rebalancing, and fault-tolerant distributed processing. While it has been mostly succeeded by the unified Beam SDKs, it remains relevant for legacy systems and offers insight into the underlying mechanisms that power scalable data workflows on Google Cloud. -
18
FLOGO
Simplify building efficient & modern serverless functions and apps
Project Flogo is an ultra-light, Go-based open source ecosystem for building event-driven apps. Event-driven, you say? Yup, the notion of triggers and actions are leveraged to process incoming events. An action, a common interface, exposes key capabilities such as application integration, stream processing, etc. All capabilities within the Flogo Ecosystem have a few things in common, they all process events (in a manner suitable for the specific purpose) and they all implement the action interface exposed by Flogo Core. Integration Flows Application Integration process engine with conditional branching and a visual development environment. A simple pipeline-based stream processing action with event joining capabilities across multiple triggers & aggregation over time windows. Microgateway pattern for conditional, content-based routing, JWT validation, rate limiting, circuit breaking and other common patterns. -
19
Fondant
Production-ready data processing made easy and shareable
Fondant is a modular, pipeline-based framework designed to simplify the preparation of large-scale datasets for training machine learning models, especially foundation models. It offers an end-to-end system for ingesting raw data, applying transformations, filtering, and formatting outputs—all while remaining scalable and traceable. Fondant is designed with reproducibility in mind and supports containerized steps using Docker, making it easy to share and reuse data processing components. It’s built for use in research and production, empowering data scientists to streamline dataset curation and preprocessing workflows efficiently. -
20
-
21
HStreamDB
HStreamDB is an open-source, cloud-native streaming database
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications. By subscribing to streams in HStreamDB, any update of the data stream will be pushed to your apps in real-time, and this promotes your apps to be more responsive. You can also replace message brokers with HStreamDB and everything you do with message brokers can be done better with HStreamDB. HStreamDB provides built-in support for event time-based stream processing. You can use your familiar SQL to perform basic filtering and transformation operations, statistics and aggregation based on multiple kinds of time windows and even joining between multiple streams. With connectors provided, you can easily integrate HStreamDB with other external systems, such as MQTT Broker, MySQL, Redis and ElasticSearch. More connectors will be added. -
22
Lithops
A multi-cloud framework for big data analytics
Lithops is an open-source serverless computing framework that enables transparent execution of Python functions across multiple cloud providers and on-prem infrastructure. It abstracts cloud providers like IBM Cloud, AWS, Azure, and Google Cloud into a unified interface and turns your Python functions into scalable, event-driven workloads. Lithops is ideal for data processing, ML inference, and embarrassingly parallel workloads, giving you the power of FaaS (Function-as-a-Service) without vendor lock-in. It also supports hybrid cloud setups, object storage access, and simple integration with Jupyter notebooks. -
23
LogsGrep
A grep-like utility for log files.
LogsGrep is a unique, grep-like utility designed specifically to target log files containing multi-line entries. The primary target is Java log files (Log4J, common, ...), where it is very common to have multiline log entries (for example log entries with a stacktrace). It follows Unix philosophy, does only its primary job and expects its input to be generated by other more advanced tools (tail, cat, type, find...); There is no goal to be compatible with Unix grep. LogsGrep is written in the Java programming langue having performance and low resource usage in mind (no strings, no object creation, stream-processing). -
24
PULSAR
Distributed pub-sub messaging system
Apache Pulsar is a cloud-native, distributed messaging and streaming platform originally created at Yahoo! and now a top-level Apache Software Foundation project. Easy to deploy, lightweight compute process, developer-friendly APIs, no need to run your own stream processing engine. Run in production at Yahoo! scale for over 5 years, with millions of messages per second across millions of topics. Expand capacity seamlessly to hundreds of nodes. Low publish latency (< 5ms) at scale with strong durability guarantees. Configurable replication between data centers across multiple geographic regions. Built from the ground up as a multi-tenant system. Supports isolation, authentication, authorization and quotas. Persistent message storage based on Apache BookKeeper. IO-level isolation between write and read operations. Flexible messaging models with high-level APIs for Java, Go, Python, C++, Node.js, WebSocket and C#. -
25
Padasip
Python Adaptive Signal Processing
Padasip (Python Adaptive Signal Processing) is a Python library tailored for adaptive filtering and online learning applications, particularly in signal processing and time series forecasting. It includes a variety of adaptive filter algorithms such as LMS, RLS, and their variants, offering real-time adaptation to changing environments. The library is lightweight, well-documented, and ideal for research, prototyping, or teaching purposes. Padasip supports both supervised and unsupervised filtering modes and is built to be modular and extensible, making it easy to integrate into larger machine learning pipelines or control systems.
Guide to Open Source Stream Processing Tools
Selecting the right open source stream processing tools begins with identifying the volume, speed, and variety of data the organization expects to process. Evaluate whether the tools support the required data sources, messaging platforms, storage technologies, and analytics environments. Consider scalability, fault tolerance, latency, and resource efficiency to ensure the solution can handle current workloads while accommodating future growth. Review deployment flexibility, operational complexity, monitoring capabilities, and security features to simplify management and maintain reliability. Strong documentation, an active development community, regular updates, and a clear licensing model can improve long-term usability and reduce maintenance challenges. Testing the tools with realistic data streams and production-like workloads helps verify performance, stability, and compatibility before making a final decision.
Features of Open Source Stream Processing Tools
- Real-time data processing: Analyzes incoming data continuously to deliver insights and actions without waiting for batch processing.
- Scalable processing: Expands resources to manage increasing data volumes while maintaining consistent performance.
- Fault tolerance: Preserves processing continuity by recovering automatically from failures or interruptions.
- Distributed architecture: Spreads workloads across multiple nodes to improve efficiency and reliability.
- Windowing functions: Groups streaming data into time-based or event-based intervals for analysis.
- Event transformation: Filters, enriches, and modifies streaming data before sending it to downstream destinations.
- Multiple data connectors: Integrates with databases, messaging platforms, cloud services, and analytics tools.
- Checkpointing support: Saves processing progress regularly to simplify recovery after unexpected disruptions.
What Types of Open Source Stream Processing Tools Are There?
- Event stream processing tools: Continuously analyze incoming events to detect patterns, trigger actions, and generate insights with minimal delay.
- Real-time analytics tools: Process live data streams to create dashboards, metrics, and reports that update as new information arrives.
- Complex event processing tools: Correlate multiple events from different sources to identify meaningful relationships and business conditions.
- Data transformation tools: Clean, enrich, filter, and reformat streaming data before forwarding it to downstream applications or storage platforms.
- Log processing tools: Collect, process, and route application, infrastructure, and security logs for monitoring and operational visibility.
- IoT stream processing tools: Handle continuous sensor data, supporting device monitoring, anomaly detection, and automated responses across connected environments.
- Messaging-based processing tools: Consume and process messages from event brokers, enabling scalable communication between distributed applications.
- Edge stream processing tools: Analyze data closer to its source, reducing latency and limiting unnecessary data transfers to centralized environments.
- Machine learning stream processing tools: Apply predictive models to live data streams for immediate scoring, recommendations, and anomaly detection.
Open Source Stream Processing Tools Benefits
- Real-time insights: Continuous data processing helps organizations identify trends, events, and operational changes as information arrives.
- Cost efficiency: Open source licensing lowers acquisition costs while allowing businesses to invest more in infrastructure and customization.
- High scalability: Distributed architectures support growing data volumes without requiring complete platform replacements.
- Flexible integration: Standard connectors enable communication with databases, messaging platforms, analytics tools, and cloud services.
- Customizable workflows: Development teams can tailor processing pipelines to meet industry requirements and unique operational goals.
- Strong reliability: Fault-tolerant designs help maintain processing continuity even when individual components experience failures.
- Active community support: Ongoing contributions improve documentation, introduce new capabilities, and address issues more rapidly.
- Deployment flexibility: Organizations can run tools across cloud, hybrid, or on-premises environments based on business and compliance needs.
- Better operational visibility: Monitoring capabilities help teams track performance, detect bottlenecks, and optimize streaming workloads.
Types of Users That Use Open Source Stream Processing Tools
- Data engineers: Build real-time data pipelines that process continuous information from multiple sources with low latency.
- IT operations teams: Monitor infrastructure events and respond quickly to changing system conditions.
- Financial institutions: Analyze transaction streams to improve visibility and support timely operational decisions.
- Telecommunications providers: Process network events continuously to maintain service quality and identify performance issues.
- Manufacturing companies: Track equipment data in real time to improve production efficiency and reduce downtime.
- Retail businesses: Monitor customer activity and inventory updates as they happen to support faster decision-making.
- Logistics providers: Process shipment and fleet data continuously to improve delivery coordination and operational visibility.
- Research teams: Evaluate streaming architectures and test new data processing techniques using customizable tools.
- Cloud architects: Design scalable streaming environments that support growing workloads across distributed infrastructure.
How Much Do Open Source Stream Processing Tools Cost?
Open source stream processing tools can often be adopted without licensing costs, making them a cost-effective choice for organizations that need to process continuous data streams. Although the tools themselves are typically free to use, businesses should plan for expenses related to infrastructure, whether they deploy on premises or in cloud environments. Overall costs depend on data volume, processing complexity, storage requirements, and the level of performance and availability needed for production workloads.
Organizations should also account for operational costs that extend beyond deployment. Implementation, configuration, monitoring, security, maintenance, and ongoing updates all require time and technical expertise. Additional investments may be necessary for integrating stream processing tools with existing data platforms, training internal teams, or developing custom workflows. Considering both infrastructure and operational expenses provides a more complete understanding of the total cost of ownership.
What Software Do Open Source Stream Processing Tools Integrate With?
Open source stream processing tools can integrate with many business technologies that support real-time data movement and analysis. Common integrations include databases, data warehouses, messaging platforms, event brokers, cloud storage services, analytics platforms, business intelligence tools, monitoring solutions, and workflow automation tools. They also connect with data integration platforms, API management solutions, machine learning platforms, and container orchestration environments to support scalable processing pipelines. Security tools, identity management solutions, and logging platforms can be integrated to strengthen governance and operational visibility. By connecting these technologies, organizations can process continuous data streams, distribute insights quickly, and improve decision-making across business operations while maintaining a flexible and extensible architecture.
Open Source Stream Processing Tools Trends
- Real-time analytics adoption continues expanding as organizations seek faster insights from continuously generated business and operational data.
- Cloud-native architectures encourage stream processing tools optimized for containerized deployments and elastic scaling.
- AI and machine learning pipelines increasingly consume streaming data to support faster predictions and automated decision-making.
- Event-driven architectures remain popular, increasing demand for reliable streaming tools that process information with minimal delay.
- Improved observability features help teams monitor throughput, latency, and processing health more effectively.
- Data governance capabilities receive greater attention, supporting compliance, auditing, and better management of streaming information.
- Hybrid and multi-cloud deployments drive demand for flexible stream processing tools operating across diverse infrastructure.
- Community collaboration accelerates new connectors, integrations, and performance improvements for broader business use.
How Users Can Get Started With Open Source Stream Processing Tools
Selecting the right open source stream processing tools begins with identifying the volume, speed, and variety of data the organization expects to process. Evaluate whether the tools support the required data sources, messaging platforms, storage technologies, and analytics environments. Consider scalability, fault tolerance, latency, and resource efficiency to ensure the solution can handle current workloads while accommodating future growth. Review deployment flexibility, operational complexity, monitoring capabilities, and security features to simplify management and maintain reliability. Strong documentation, an active development community, regular updates, and a clear licensing model can improve long-term usability and reduce maintenance challenges. Testing the tools with realistic data streams and production-like workloads helps verify performance, stability, and compatibility before making a final decision.
