Search Results for "data processing"
Sort By:
Showing 28 open source projects for "data processing"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
$300 in Free Credit Towards Top Cloud ServicesStart your project in minutes. After credits run out, 20+ products include free monthly usage. Only pay when you're ready to scale.
-
1
Kapacitor
Open source framework for processing, monitoring, and alerting
Open source framework for processing, monitoring, and alerting on time series data. Kapacitor is a real-time data processing engine for monitoring and alerting, specifically designed to work with time-series data from InfluxDB. -
2
Numaflow
Kubernetes-native platform to run massively parallel data/streaming
Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices. Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platform. -
3
Pachyderm
Data-Centric Pipelines and Data Versioning
...Pachyderm provides a powerful solution to optimize data processing, MLOps, and ML Lifecycles. -
4
Benthos
Fancy stream processing made operationally mundane
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads. It comes with a powerful mapping language, is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary, docker image, or serverless function, making it cloud native as heck. Delivery guarantees can be a dodgy subject. Benthos processes and... -
Add Two Lines of Code. Get Full APM.Works out of the box for Rails, Django, Express, Phoenix, and more. Monitoring exceptions and performance in no time.
-
5
Watermill
Building event-driven applications the easy way in Go
Go library for building event-driven applications. Our goal was to create a tool that is easy to understand, even by junior developers. It doesn't matter if you want to do Event-driven architecture, CQRS, Event Sourcing or just stream MySQL Binlog to Kafka. Watermill was designed to process hundreds of thousands of messages per second. Every component is built in a way that allows you to configure it for your needs. You can also implement your own middleware for the router. Watermill is... -
6
protoactor-go
Proto Actor - Ultra fast distributed actors for Go, C# and Java/Kotlin
Built on cloud-native technologies. Taking advantage of proven stability and performance. Asynchronous and Distributed by design. High-level abstractions like Actors and Virtual Grains. Capable of millions of messages per second cross-process communication. Write systems that self-heal using supervisor hierarchies. The Actor Model provides a higher level of abstraction for writing concurrent and distributed systems. It alleviates the developer from having to deal with explicit locking and... -
7
Nuclio
High-Performance Serverless event and data processing platform
Nuclio is an open source and managed serverless platform used to minimize development and maintenance overhead and automate the deployment of data-science-based applications. Real-time performance running up to 400,000 function invocations per second. Portable across low laptops, edge, on-prem and multi-cloud deployments. The first serverless platform supporting GPUs for optimized utilization and sharing. Automated deployment to production in a few clicks from Jupyter notebook. Deploy one of... -
8
Hacks
A collection of hacks and one-off scripts
Hacks is a collection of experimental scripts, utilities, and one-off tools created to solve specific problems in security research, data processing, and automation. Rather than being a single cohesive application, it serves as a repository of practical command-line tools that can be used independently or combined into workflows. The scripts cover a wide range of tasks, including URL manipulation, parameter replacement, data extraction, and reconnaissance automation. ... -
9
Bacalhau
Community-driven, simple, yet powerful framework
Bacalhau is a decentralized compute platform for running jobs on data stored across distributed networks, like IPFS or Filecoin, without moving the data to centralized cloud environments. It allows developers to run containerized workloads close to where the data lives, reducing latency, cost, and privacy risks. Bacalhau supports various runtime environments and is designed to make decentralized data processing as accessible as traditional cloud computing. ... -
Full-stack observability with actually useful AI | Grafana CloudBuilt on open standards like Prometheus and OpenTelemetry, Grafana Cloud includes Kubernetes Monitoring, Application Observability, Incident Response, plus the AI-powered Grafana Assistant. Get started with our generous free tier today.
-
10
Colly
Elegant Scraper and Crawler Framework for Golang
Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling. Sync/async/parallel scraping. Distributed scraping. Caching, automatic encoding of non-unicode responses. ... -
11
Argo Workflows
Workflow engine for Kubernetes
...Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a directed acyclic graph (DAG). Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes. Run CI/CD pipelines natively on Kubernetes without configuring complex software development products. Argo Workflows is the most popular workflow execution engine for Kubernetes. It can run 1000s of workflows a day, each with 1000s of concurrent tasks. Our users say it is lighter-weight, faster, more powerful, and easier to use. ... -
12
Loggie
A lightweight, cloud-native data transfer agent and aggregator
Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang. Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation. Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources. Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3. ... -
13
RuleGo
Component orchestration rule engine framework for Go
...It’s lightweight, embeddable, orchestration-ready, and built for flexible composition of business logic into reusable components. No external middleware dependencies, efficient data processing and linkage on low-cost devices, suitable for IoT edge computing. Embedded and Standalone Deployment modes. Supports embedding RuleGo into existing applications. It can also be deployed independently as middleware, providing rule engine and orchestration services. -
14
XLSX
Go (golang) library for reading and writing XLSX files
...These can be used to modify the behavior of the resultant struct, in particular they replace the `…WithRowLimit` variants of those methods with the result of calling `xlsx.RowLimit` and they add the ability to define a custom backing store for the spreadsheet data to be held in whilst processing. The full API docs can be viewed using go’s built in documentation tool. -
15
KubeEdge
Kubernetes Native Edge Computing Framework (project under CNCF)
...It also supports MQTT which enables edge devices to access through edge nodes. With KubeEdge it is easy to get and deploy existing complicated machine learning, image recognition, event processing, and other high-level applications to the Edge. With business logic running at the Edge, much larger volumes of data can be secured & processed locally where the data is produced. With data processed at the Edge, the responsiveness is increased dramatically and data privacy is protected. -
16
ergo
Framework for creating microservices using technologies of Erlang/OTP
...The goal of this project is to leverage Erlang/OTP experience with Golang performance. The ideal framework for creating complex and distributed solutions (machine learning, data processing pipeline, etc.) being simple and reliable. You don't have to reinvent the wheel. There are ready-to-use implemented design patterns. Two processes can be linked to each other. Termination one terminates another. Any process can monitor the service node. Receives NODE DOWN if node terminated. Ergo Framework almost 5 times outperforms the original Erlang network messaging. ... -
17
ldetool
Code generator for fast log file parsers
ldetool (Line Data Extraction Tool) is a command-line utility that generates Go code for fast log file parsing. By defining parsing rules, developers can produce efficient parsers tailored to specific log formats, outperforming traditional regex-based approaches. It's particularly useful for processing large volumes of log data. -
18
git-chglog
CHANGELOG generator implemented in Go (Golang)
...Automatic linking of issues and notices, initial value setting such as merges etc. are done automatically. Metadata for CHANGELOG. Depending on Style, it is sometimes used in processing, so it is recommended to specify it. -
19
go-coffeeshop
A practical event-driven microservices demo
...It also includes examples of integrating external systems and handling real-world scenarios such as order processing and inventory management. -
20
QOR
Libraries written in Go abstracting features needed for business apps
QOR is architected from the ground up to accelerate development and deployment of Content Management Systems, E-commerce Systems, and Business Applications. QOR is comprised of modules that abstract common features for such systems, such as a configurable and flexible back office, a content publishing system, a media library, and much more. We built QOR at The Plant to accelerate development for our clients, it is at the core of our engineering strategy, and we maintain it on an ongoing... -
21
kube-fledged
Kubernetes operator to create and manage cache of container images
kube-fledged is a kubernetes operator for creating and managing a cache of container images directly on the worker nodes of a Kubernetes cluster. It allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pulled). As a result, application pods start almost instantly, since the images need not be pulled from the registry. kube-fledged provides CRUD APIs to manage the lifecycle of the image cache, and supports several configurable parameters to... -
22
Brigade
Event-driven scripting for Kubernetes
Brigade is a full-featured, event-driven scripting platform built on top of Kubernetes. It integrates with many different event sources, more are always being added, and it's easy to create your own if you need something specific. The best part is that Kubernetes is well-abstracted so even team members without extensive Kubernetes experience or without direct access to a cluster can be productive. -
23
Revel
A high productivity, full-stack web framework for the Go language
Edit, save, and refresh. Revel compiles your code and templates for you, so you don't miss a beat. Code doesn't compile? It gives you a helpful description. Run-time code panic? Revel has you covered. Revel provides routing, parameter parsing, validation, session/flash, templating, caching, job running, a testing framework, and even internationalization. Revel is built around composable middleware called filters, which implement nearly all request-processing functionality. Developers have... -
24
RxGo
Reactive Extensions for the Go language
...This is the official ReactiveX API for the Go language. ReactiveX is a new, alternative way of asynchronous programming to callbacks, promises, and deferred. It is about processing streams of events or items, with events being any occurrences or changes within the system. A stream of events is called an Observable. An operator is a function that defines an Observable, how and when it should emit data. The RxGo implementation is based on the concept of pipelines. A pipeline is a series of stages connected by channels, where each stage is a group of goroutines running the same function. ... -
25
FLOGO
Simplify building efficient & modern serverless functions and apps
Project Flogo is an ultra-light, Go-based open source ecosystem for building event-driven apps. Event-driven, you say? Yup, the notion of triggers and actions are leveraged to process incoming events. An action, a common interface, exposes key capabilities such as application integration, stream processing, etc. All capabilities within the Flogo Ecosystem have a few things in common, they all process events (in a manner suitable for the specific purpose) and they all implement the action...
