Search Results for "data processing"
Sort By:
Showing 29 open source projects for "data processing"
View related business solutions-
Earn up to 16% annual interest with Nexo.Generate interest, borrow against your crypto, and trade a range of cryptocurrencies — all in one platform. Geographic restrictions, eligibility, and terms apply.
-
One Monitoring Tool for IT, OT and Cloud | Free TrialModern infrastructure spans data centers, cloud platforms and factory floors, and every blind spot between them is a risk. PRTG supports SNMP, WMI, SSH and other standard protocols to monitor IT, OT and hybrid environments through one customizable dashboard. Build the views your team needs, from network health to application performance, without switching tools. Try PRTG free for 30 days now.
-
1
Kapacitor
Open source framework for processing, monitoring, and alerting
Open source framework for processing, monitoring, and alerting on time series data. Kapacitor is a real-time data processing engine for monitoring and alerting, specifically designed to work with time-series data from InfluxDB. -
2
Numaflow
Kubernetes-native platform to run massively parallel data/streaming
Numaflow is a Kubernetes-native tool for running massively parallel stream processing. A Numaflow Pipeline is implemented as a Kubernetes custom resource and consists of one or more source, data processing, and sink vertices. Numaflow installs in a few minutes and is easier and cheaper to use for simple data processing applications than a full-featured stream processing platform. -
3
Pachyderm
Data-Centric Pipelines and Data Versioning
...Pachyderm provides a powerful solution to optimize data processing, MLOps, and ML Lifecycles. -
4
Hacks
A collection of hacks and one-off scripts
Hacks is a collection of experimental scripts, utilities, and one-off tools created to solve specific problems in security research, data processing, and automation. Rather than being a single cohesive application, it serves as a repository of practical command-line tools that can be used independently or combined into workflows. The scripts cover a wide range of tasks, including URL manipulation, parameter replacement, data extraction, and reconnaissance automation. ... -
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
5
Benthos
Fancy stream processing made operationally mundane
Benthos is a high performance and resilient stream processor, able to connect various sources and sinks in a range of brokering patterns and perform hydration, enrichments, transformations and filters on payloads. It comes with a powerful mapping language, is easy to deploy and monitor, and ready to drop into your pipeline either as a static binary, docker image, or serverless function, making it cloud native as heck. Delivery guarantees can be a dodgy subject. Benthos processes and... -
6
Watermill
Building event-driven applications the easy way in Go
Go library for building event-driven applications. Our goal was to create a tool that is easy to understand, even by junior developers. It doesn't matter if you want to do Event-driven architecture, CQRS, Event Sourcing or just stream MySQL Binlog to Kafka. Watermill was designed to process hundreds of thousands of messages per second. Every component is built in a way that allows you to configure it for your needs. You can also implement your own middleware for the router. Watermill is... -
7
Bacalhau
Community-driven, simple, yet powerful framework
Bacalhau is a decentralized compute platform for running jobs on data stored across distributed networks, like IPFS or Filecoin, without moving the data to centralized cloud environments. It allows developers to run containerized workloads close to where the data lives, reducing latency, cost, and privacy risks. Bacalhau supports various runtime environments and is designed to make decentralized data processing as accessible as traditional cloud computing. ... -
8
gitcrawl
Local-first GitHub issue and pull request crawler
gitcrawl is a repository crawling and indexing tool associated with the OpenClaw ecosystem, designed to collect, organize, and analyze Git-based project data at scale. The project focuses on automating repository inspection and enabling structured access to source code, commit history, and metadata for research, automation, or AI-assisted workflows. Its architecture likely emphasizes efficient crawling, local indexing, and integration with other OpenClaw tooling for autonomous analysis and... -
9
Nuclio
High-Performance Serverless event and data processing platform
Nuclio is an open source and managed serverless platform used to minimize development and maintenance overhead and automate the deployment of data-science-based applications. Real-time performance running up to 400,000 function invocations per second. Portable across low laptops, edge, on-prem and multi-cloud deployments. The first serverless platform supporting GPUs for optimized utilization and sharing. Automated deployment to production in a few clicks from Jupyter notebook. Deploy one of... -
Save Up to 91% on Cloud Compute With Spot VMsRun batch jobs at 60-91% off with Spot VMs. Long-running workloads get automatic discounts with sustained use.
-
10
Colly
Elegant Scraper and Crawler Framework for Golang
Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving. Clean API. Fast (>1k request/sec on a single core) Manages request delays and maximum concurrency per domain. Automatic cookie and session handling. Sync/async/parallel scraping. Distributed scraping. Caching, automatic encoding of non-unicode responses. ... -
11
Argo Workflows
Workflow engine for Kubernetes
...Model multi-step workflows as a sequence of tasks or capture the dependencies between tasks using a directed acyclic graph (DAG). Easily run compute intensive jobs for machine learning or data processing in a fraction of the time using Argo Workflows on Kubernetes. Run CI/CD pipelines natively on Kubernetes without configuring complex software development products. Argo Workflows is the most popular workflow execution engine for Kubernetes. It can run 1000s of workflows a day, each with 1000s of concurrent tasks. Our users say it is lighter-weight, faster, more powerful, and easier to use. ... -
12
XLSX
Go (golang) library for reading and writing XLSX files
...These can be used to modify the behavior of the resultant struct, in particular they replace the `…WithRowLimit` variants of those methods with the result of calling `xlsx.RowLimit` and they add the ability to define a custom backing store for the spreadsheet data to be held in whilst processing. The full API docs can be viewed using go’s built in documentation tool. -
13
RuleGo
Component orchestration rule engine framework for Go
...It’s lightweight, embeddable, orchestration-ready, and built for flexible composition of business logic into reusable components. No external middleware dependencies, efficient data processing and linkage on low-cost devices, suitable for IoT edge computing. Embedded and Standalone Deployment modes. Supports embedding RuleGo into existing applications. It can also be deployed independently as middleware, providing rule engine and orchestration services. -
14
kube-fledged
Kubernetes operator to create and manage cache of container images
kube-fledged is a kubernetes operator for creating and managing a cache of container images directly on the worker nodes of a Kubernetes cluster. It allows a user to define a list of images and onto which worker nodes those images should be cached (i.e. pulled). As a result, application pods start almost instantly, since the images need not be pulled from the registry. kube-fledged provides CRUD APIs to manage the lifecycle of the image cache, and supports several configurable parameters to... -
15
KubeEdge
Kubernetes Native Edge Computing Framework (project under CNCF)
...It also supports MQTT which enables edge devices to access through edge nodes. With KubeEdge it is easy to get and deploy existing complicated machine learning, image recognition, event processing, and other high-level applications to the Edge. With business logic running at the Edge, much larger volumes of data can be secured & processed locally where the data is produced. With data processed at the Edge, the responsiveness is increased dramatically and data privacy is protected. -
16
ergo
Framework for creating microservices using technologies of Erlang/OTP
...The goal of this project is to leverage Erlang/OTP experience with Golang performance. The ideal framework for creating complex and distributed solutions (machine learning, data processing pipeline, etc.) being simple and reliable. You don't have to reinvent the wheel. There are ready-to-use implemented design patterns. Two processes can be linked to each other. Termination one terminates another. Any process can monitor the service node. Receives NODE DOWN if node terminated. Ergo Framework almost 5 times outperforms the original Erlang network messaging. ... -
17
protoactor-go
Proto Actor - Ultra fast distributed actors for Go, C# and Java/Kotlin
Built on cloud-native technologies. Taking advantage of proven stability and performance. Asynchronous and Distributed by design. High-level abstractions like Actors and Virtual Grains. Capable of millions of messages per second cross-process communication. Write systems that self-heal using supervisor hierarchies. The Actor Model provides a higher level of abstraction for writing concurrent and distributed systems. It alleviates the developer from having to deal with explicit locking and... -
18
Loggie
A lightweight, cloud-native data transfer agent and aggregator
Loggie is a lightweight, high-performance, cloud-native agent and aggregator based on Golang. Loggie includes LogConfig/ClusterLogConfig/Interceptor/Sink CRDs, allowing for the creation of data collection, transfer, processing, and sending pipelines through simple YAML file creation. Supports deployment as an independent intermediate machine, which can receive aggregated data sent by Loggie Agent and can also be used to consume and process various data sources. Configure Filebeat and Loggie to collect logs, and send them to a Kafka topic without using client compression, with the Kafka topic partition configured as 3. ... -
19
ldetool
Code generator for fast log file parsers
ldetool (Line Data Extraction Tool) is a command-line utility that generates Go code for fast log file parsing. By defining parsing rules, developers can produce efficient parsers tailored to specific log formats, outperforming traditional regex-based approaches. It's particularly useful for processing large volumes of log data. -
20
git-chglog
CHANGELOG generator implemented in Go (Golang)
...Automatic linking of issues and notices, initial value setting such as merges etc. are done automatically. Metadata for CHANGELOG. Depending on Style, it is sometimes used in processing, so it is recommended to specify it. -
21
go-coffeeshop
A practical event-driven microservices demo
...It also includes examples of integrating external systems and handling real-world scenarios such as order processing and inventory management. -
22
QOR
Libraries written in Go abstracting features needed for business apps
QOR is architected from the ground up to accelerate development and deployment of Content Management Systems, E-commerce Systems, and Business Applications. QOR is comprised of modules that abstract common features for such systems, such as a configurable and flexible back office, a content publishing system, a media library, and much more. We built QOR at The Plant to accelerate development for our clients, it is at the core of our engineering strategy, and we maintain it on an ongoing... -
23
Brigade
Event-driven scripting for Kubernetes
Brigade is a full-featured, event-driven scripting platform built on top of Kubernetes. It integrates with many different event sources, more are always being added, and it's easy to create your own if you need something specific. The best part is that Kubernetes is well-abstracted so even team members without extensive Kubernetes experience or without direct access to a cluster can be productive. -
24
Revel
A high productivity, full-stack web framework for the Go language
Edit, save, and refresh. Revel compiles your code and templates for you, so you don't miss a beat. Code doesn't compile? It gives you a helpful description. Run-time code panic? Revel has you covered. Revel provides routing, parameter parsing, validation, session/flash, templating, caching, job running, a testing framework, and even internationalization. Revel is built around composable middleware called filters, which implement nearly all request-processing functionality. Developers have... -
25
RxGo
Reactive Extensions for the Go language
...This is the official ReactiveX API for the Go language. ReactiveX is a new, alternative way of asynchronous programming to callbacks, promises, and deferred. It is about processing streams of events or items, with events being any occurrences or changes within the system. A stream of events is called an Observable. An operator is a function that defines an Observable, how and when it should emit data. The RxGo implementation is based on the concept of pipelines. A pipeline is a series of stages connected by channels, where each stage is a group of goroutines running the same function. ...
