Open Source Linux Stream Processing Tools - Page 2
Sort By:
Stream Processing Tools for Linux
View 8 business solutions-
$300 Free Credits for Your Google Cloud ProjectsLaunch your next project with $300 in free Google Cloud credits—no strings attached. Test, build, and deploy without risk. Use your credits across the entire Google Cloud platform to find what works best for your needs. After your credits are used, continue with always-free tier services. Only pay when you're ready to scale. Sign up in minutes and start exploring.
-
Enterprise-grade ITSM, for every businessFreshservice is an intuitive, AI-powered platform that helps IT, operations, and business teams deliver exceptional service without the usual complexity. Automate repetitive tasks, resolve issues faster, and provide seamless support across the organization. From managing incidents and assets to driving smarter decisions, Freshservice makes it easy to stay efficient and scale with confidence.
-
1
An experimental CEP (Complex Event Processing) engine. It implements the event stream processing as a library embeddable in C++ and Perl. Since then it has been renamed to Triceps, so please look at the new location https://sourceforge.net/projects/t
-
2
Cosmos DB Spark
Apache Spark Connector for Azure Cosmos DB
Azure Cosmos DB Spark is the official connector for Azure CosmosDB and Apache Spark. The connector allows you to easily read to and write from Azure Cosmos DB via Apache Spark DataFrames in Python and Scala. It also allows you to easily create a lambda architecture for batch-processing, stream-processing, and a serving layer while being globally replicated and minimizing the latency involved in working with big data. -
3
DSPatch
The Refreshingly Simple C++ Dataflow Framework
Webite: http://flowbasedprogramming.com DSPatch, pronounced "dispatch", is a powerful C++ dataflow framework. DSPatch is not limited to any particular domain or data type, from reactive programming to stream processing, DSPatch's generic, object-oriented API allows you to create virtually any dataflow system imaginable. *See also:* DSPatcher ( https://github.com/MarcusTomlinson/DSPatcher ): A cross-platform graphical tool for building DSPatch circuits. DSPatchables ( https://github.com/MarcusTomlinson/DSPatchables ): A DSPatch component repository. -
4
Dataflow Java SDK
Google Cloud Dataflow provides a simple, powerful model
The Dataflow Java SDK is the open-source Java library that powers Apache Beam pipelines for Google Cloud Dataflow, a serverless and scalable platform for processing large datasets in both batch and stream modes. This SDK allows developers to write Beam-based pipelines in Java and execute them on Dataflow, taking advantage of features like autoscaling, dynamic work rebalancing, and fault-tolerant distributed processing. While it has been mostly succeeded by the unified Beam SDKs, it remains relevant for legacy systems and offers insight into the underlying mechanisms that power scalable data workflows on Google Cloud. -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Fondant
Production-ready data processing made easy and shareable
Fondant is a modular, pipeline-based framework designed to simplify the preparation of large-scale datasets for training machine learning models, especially foundation models. It offers an end-to-end system for ingesting raw data, applying transformations, filtering, and formatting outputs—all while remaining scalable and traceable. Fondant is designed with reproducibility in mind and supports containerized steps using Docker, making it easy to share and reuse data processing components. It’s built for use in research and production, empowering data scientists to streamline dataset curation and preprocessing workflows efficiently. -
6
-
7
HStreamDB
HStreamDB is an open-source, cloud-native streaming database
HStreamDB is an open-source, cloud-native streaming database for IoT and beyond. Modernize your data stack for real-time applications. By subscribing to streams in HStreamDB, any update of the data stream will be pushed to your apps in real-time, and this promotes your apps to be more responsive. You can also replace message brokers with HStreamDB and everything you do with message brokers can be done better with HStreamDB. HStreamDB provides built-in support for event time-based stream processing. You can use your familiar SQL to perform basic filtering and transformation operations, statistics and aggregation based on multiple kinds of time windows and even joining between multiple streams. With connectors provided, you can easily integrate HStreamDB with other external systems, such as MQTT Broker, MySQL, Redis and ElasticSearch. More connectors will be added. -
8
LogsGrep
A grep-like utility for log files.
LogsGrep is a unique, grep-like utility designed specifically to target log files containing multi-line entries. The primary target is Java log files (Log4J, common, ...), where it is very common to have multiline log entries (for example log entries with a stacktrace). It follows Unix philosophy, does only its primary job and expects its input to be generated by other more advanced tools (tail, cat, type, find...); There is no goal to be compatible with Unix grep. LogsGrep is written in the Java programming langue having performance and low resource usage in mind (no strings, no object creation, stream-processing). -
9
MXQuery is a low-footprint implementation of XQuery 1.0, XQuery Update 1.0, XQuery Fulltext 1.0 and XQuery Scripting 1.0 as well as a subset of XQuery 1.1 (windowing, try/catch). It provides extensions to do data stream processing/CEP and SOAP/REST
-
Error to trace to log to deploy. One click. No SSH.AppSignal links every error to the trace, the trace to the log, the log to the deploy that shipped it.
-
10
Sed.py is a python module to provide a easy way to do text stream processing. Just like the name of module, it likes to do the work that sed can do. But not in sed's way, it's in Python's way. To use this module, the knowledge of regexp is necessary.
-
11
A production stable Java utility library with convenience methods for string- and stream processing, file handling, XML, XSLTs and XPath, checksums, console formatting, and more. The project is developed by the State and University Library of Denmark
-
12
SPar: Stream Parallelism in Multi-Cores
An Embedded C++ Domain-Specific Language
SPar is an internal C++ Domain-Specific Language (DSL) suitable to model and implement classical stream parallel patterns. The DSL uses standard C++ attributes to introduce annotations tagging the notable components of stream parallel applications: stream sources and stream processing stages. Latest version can be downloaded from the SVN using the following command: svn checkout svn://svn.code.sf.net/p/spar-dsl-compiler/svn/ spar -
13
Strings Edit
String editing and formatting library for Ada
Strings edit is a library that provides I/O facilities for integers, floating-point numbers, Roman numerals, and strings. Both input and output subroutines support string pointers for consequent stream processing. The output can be aligned in a fixed size field with padding. Numeric input can be checked against expected values range to be either saturated or to raise an exception. For floating-point output either relative or absolute output precision can be specified. UTF-8 encoded strings are supported, including wildcard pattern matching, sets and maps of code points, upper/lowercase, and other Unicode categorizations. -
14
An innovative Open Source CEP (Complex Event Processing) engine. It implements the event stream processing as a library embeddable in C++ and Perl. You can think of the Complex Event Processing engine as an in-memory database driven by triggers, or a data-flow machine, or a spreadsheet on steroids (and without the GUI part).
-
15
ActiveInsight provides real-time detection and reaction to events and patterns. It is a platform that enables the detection of meaningful events within multiple, high frequency, event streams.
-
16
collapse
Advanced and Fast Data Transformation in R
collapse is a high-performance R package designed for fast and efficient data transformation, aggregation, reshaping, and statistical computation. Built to offer a more performant alternative to dplyr and data.table, it is particularly well-suited for large datasets and econometric applications. It operates on base R data structures like data frames and vectors and uses highly optimized C++ code under the hood to deliver significant speed improvements. collapse also includes tools for grouped operations, weighted statistics, and time series manipulation, making it a compact yet powerful utility for data scientists and researchers working in R. -
17
horizon
Horizon is a realtime, open-source backend for JavaScript apps
Horizon is an open-source developer platform for building sophisticated realtime apps. It provides a complete backend that makes it dramatically simpler to build, deploy, manage, and scale engaging JavaScript web and mobile apps. Horizon is extensible, integrates with the Node.js stack, and allows building modern, arbitrarily complex applications. While technologies like RethinkDB and WebSocket make it possible to build engaging realtime apps, empirically there is still too much friction for most developers. Building realtime apps now requires understanding and manually orchestrating multiple systems across the software stack, understanding distributed stream processing, and learning how to deploy and scale realtime systems. The learning curve is quite steep, and most of the initial work involves boilerplate code that is far removed from the primary task of building a realtime app. -
18
ksqlDB
The database purpose-built for stream processing applications
Build applications that respond immediately to events. Craft materialized views over streams. Receive real-time push updates, or pull current state on demand. Seamlessly leverage your existing Apache Kafka® infrastructure to deploy stream-processing workloads and bring powerful new capabilities to your applications. Use a familiar, lightweight syntax to pack a powerful punch. Capture, process, and serve queries using only SQL. No other languages or services are required. ksqlDB enables you to build event streaming applications leveraging your familiarity with relational databases. Three categories are foundational to building an application: collections, stream processing, and queries. Streams are immutable, append-only sequences of events. They're useful for representing a series of historical facts. Tables are mutable collections of events. They let you represent the latest version of each value per key. -
19
text-dedup
All-in-one text de-duplication
text-dedup is a Python library that enables efficient deduplication of large text corpora by using MinHash and other probabilistic techniques to detect near-duplicate content. This is especially useful for NLP tasks where duplicated training data can skew model performance. text-dedup scales to billions of documents and offers tools for chunking, hashing, and comparing text efficiently with low memory usage. It supports Jaccard similarity thresholding, parallel execution, and flexible deduplication strategies, making it ideal for cleaning web-scraped data, language model training datasets, or document archives.
