Search Results for "cluster computing"
Sort By:
Showing 81 open source projects for "cluster computing"
View related business solutions-
$300 Free Credits to Build on Google CloudPut your $300 in credit toward real workloads, then keep building with free monthly usage for 20+ products. No commitment and no charge until you upgrade.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
Kraken
P2P Docker registry capable of distributing TBs of data in seconds
Kraken is a P2P-powered Docker registry that focuses on scalability and availability. It is designed for Docker image management, replication, and distribution in a hybrid cloud environment. With pluggable backend support, Kraken can easily integrate into existing Docker registry setups as the distribution layer. Kraken has been in production at Uber since early 2018. In our busiest cluster, Kraken distributes more than 1 million blobs per day, including 100k 1G+ blobs. At its peak... -
2
Apache Sedona
Cluster computing framework for processing large-scale geospatial data
Apache Sedona™ is a cluster computing system for processing large-scale spatial data. Sedona extends existing cluster computing systems, such as Apache Spark and Apache Flink, with a set of out-of-the-box distributed Spatial Datasets and Spatial SQL that efficiently load, process, and analyze large-scale spatial data across machines. According to our benchmark and third-party research papers, Sedona runs 2X - 10X faster than other Spark-based geospatial data systems on computation-intensive query workloads. ... -
3
Dagger.jl
A framework for out-of-core and parallel execution
Dagger.jl is a framework for out-of-core and parallel computing in Julia that allows users to construct and execute dynamic task graphs. It is designed for large-scale, distributed, and memory-efficient computations. Dagger supports lazy evaluation and scheduling across multiple threads or machines, enabling high-performance workflows for data processing, scientific computing, and machine learning. -
4
LINKERD
Ultralight, security-first service mesh for Kubernetes
Enterprise power without enterprise complexity. Linkerd adds security, observability, and reliability to any Kubernetes cluster. 100% open source, CNCF graduated, and written in Rust. Instantly add latency-aware load balancing, request retries, timeouts, and blue-green deploys to keep your applications resilient. Incredibly small and blazing fast Linkerd2-proxy micro-proxy written in Rust for security and performance. Self-contained control plane, incrementally deployable data plane, and... -
Build Securely on Azure with Proven FrameworksMoving to the cloud brings new challenges. How can you manage a larger attack surface while ensuring great network performance? Turn to Fortinet’s Tested Reference Architectures, blueprints for designing and securing cloud environments built by cybersecurity experts. Learn more and explore use cases in this white paper.
-
5
kubeinvaders
Gamified Chaos Engineering Tool for Kubernetes
This project is part of the landscape of Cloud Native Computing Foundation in the Observability and Analysis - Chaos Engineering section. Backed by the teams at platform engineering.it and devopstribe.it, which provides enterprise-grade features and certified resilience services for your Kubernetes infrastructure. -
6
sparklyr
R interface for Apache Spark
sparklyr is an R package that provides seamless interfacing with Apache Spark clusters—either local or remote—while letting users write code in familiar R paradigms. It supplies a dplyr-compatible backend, Spark machine learning pipelines, SQL integration, and I/O utilities to manipulate and analyze large datasets distributed across cluster environments. -
7
tsuru
Open source and extensible Platform as a Service (PaaS)
Tsuru is an open source Platform as a Service software, focused in Developer productivity. Tsuru goes beyond 12 factor apps. Run any application written in any language or framework. The deploy process is really simple with just one command. Grow your application dynamically allocating resources with ease. Let your developers code and understand the business instead of solving infrastructure problems or handling large configuration files. We built the platform as a service atop of... -
8
Kubernetes Handbook
Cloud native application architecture practice handbook
...Kubernetes is a container orchestration and scheduling engine developed by Google in June 2014 based on its internal Borg system. Google contributed it as an initial and core project to the CNCF (Cloud Native Computing Foundation). In recent years, it has gradually developed a cloud native Ecology. The goal of Kubernetes is to provide a specification to describe the architecture of the cluster. -
9
MicroK8s
Single-package Kubernetes for developers, IoT and edge
Low-ops, minimal production Kubernetes, for devs, cloud, clusters, workstations, Edge and IoT. MicroK8s automatically chooses the best nodes for the Kubernetes datastore. When you lose a cluster database node, another node is promoted. No admin needed for your bulletproof edge. MicroK8s is small, with sensible defaults that ‘just work’. A quick install, easy upgrades and great security make it perfect for micro clouds and edge computing. As the publishers of MicroK8s, we deliver the world’s most efficient multi-cloud, multi-arch Kubernetes. ... -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
10
Redis
An in-memory database that persists on disk
...Redis has built-in replication, Lua scripting, LRU eviction, transactions, and different levels of on-disk persistence, and provides high availability via Redis Sentinel and automatic partitioning with Redis Cluster. You can run atomic operations on these types, like appending to a string; incrementing the value in a hash; pushing an element to a list; computing set intersection, union and difference; or getting the member with highest ranking in a sorted set. To achieve top performance, Redis works with an in-memory dataset. Depending on your use case, you can persist your data either by periodically dumping the dataset to disk or by appending each command to a disk-based log. -
11
Covalent workflow
Pythonic tool for running machine-learning/high performance workflows
Covalent is a Pythonic workflow tool for computational scientists, AI/ML software engineers, and anyone who needs to run experiments on limited or expensive computing resources including quantum computers, HPC clusters, GPU arrays, and cloud services. Covalent enables a researcher to run computation tasks on an advanced hardware platform – such as a quantum computer or serverless HPC cluster – using a single line of code. Covalent overcomes computational and operational challenges inherent in AI/ML experimentation. -
12
Kubeflow Trainer
Distributed AI Model Training and LLM Fine-Tuning on Kubernetes
...The platform supports a wide range of machine learning frameworks, including PyTorch, JAX, Hugging Face, DeepSpeed, and XGBoost, making it highly flexible for different AI use cases. One of its key innovations is the integration of MPI-based distributed computing within Kubernetes, allowing efficient communication between nodes for high-performance training. It also includes advanced scheduling capabilities through integrations with tools like Kueue and Volcano, enabling topology-aware resource allocation and multi-cluster job orchestration. -
13
Apache Seata
High-performance, open source distributed transaction solution
Seata is a distributed transaction solution for microservices that provides consistent, cross-service commits without forcing every team to adopt the same persistence model. Its architecture separates responsibilities into a global coordinator and per-service participants, so business services remain decoupled while transactions are orchestrated centrally. Multiple modes are supported—AT (automatic, SQL-based with undo logs), TCC (try-confirm-cancel), Saga (long-running compensation), and... -
14
PelicanHPC is an iso-hybrid (CD or USB) image that let's you set up a high performance computing cluster in a few minutes. A Pelican cluster allows you to do parallel computing using MPI. You can run Pelican on a single multiple core machine to use all cores to solve a problem, or you can network multiple computers together to make a cluster. The frontend node (either a real computer or a virtual machine) boots from the image.
-
15
AlphaZero.jl
A generic, simple and fast implementation of Deepmind's AlphaZero
Beyond its much publicized success in attaining superhuman level at games such as Chess and Go, DeepMind's AlphaZero algorithm illustrates a more general methodology of combining learning and search to explore large combinatorial spaces effectively. We believe that this methodology can have exciting applications in many different research areas. Because AlphaZero is resource-hungry, successful open-source implementations (such as Leela Zero) are written in low-level languages (such as C++)... -
16
AWS ParallelCluster Cookbook
The Chef cookbook used to build and bootstrap AWS ParallelCluster
AWS ParallelCluster is an AWS supported Open Source cluster management tool that makes it easy for you to deploy and manage High Performance Computing (HPC) clusters in the AWS cloud. Built on the Open Source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and a shared filesystem and offers a variety of batch schedulers such as AWS Batch and Slurm. -
17
Smile
Statistical machine intelligence and learning engine
Smile is a fast and comprehensive machine learning engine. With advanced data structures and algorithms, Smile delivers the state-of-art performance. Compared to this third-party benchmark, Smile outperforms R, Python, Spark, H2O, xgboost significantly. Smile is a couple of times faster than the closest competitor. The memory usage is also very efficient. If we can train advanced machine learning models on a PC, why buy a cluster? Write applications quickly in Java, Scala, or any JVM... -
18
AWS ParallelCluster Node
Python package installed on the Amazon EC2 instances
aws-parallelcluster-node is the python package installed on the Amazon EC2 instances launched as part of AWS ParallelCluster. AWS ParallelCluster is an AWS-supported Open Source cluster management tool that makes it easy for you to deploy and manage High-Performance Computing (HPC) clusters in the AWS cloud. Built on the Open Source CfnCluster project, AWS ParallelCluster enables you to quickly build an HPC compute environment in AWS. It automatically sets up the required compute resources and a shared filesystem and offers a variety of batch schedulers such as AWS Batch and Slurm. ... -
19
Matrix
Multi-Agent daTa geneRation Infra and eXperimentation framework
Matrix is a distributed, large-scale engine for multi-agent synthetic data generation and experiments: it provides the infrastructure to run thousands of “agentic” workflows concurrently (e.g. multiple LLMs interacting, reasoning, generating content, data-processing pipelines) by leveraging distributed computing (like Ray + cluster management). The idea is to treat data generation as a “data-to-data” transformation: each input item defines a task, and the runtime orchestrates asynchronous, peer-to-peer agent workflows, avoiding global synchronization bottlenecks. That design makes Matrix particularly well-suited for large-batch inference, model benchmarking, data curation, augmentation, or generation — whether for language, code, dialogue, or multimodal tasks. ... -
20
KubeOperator
An open source, lightweight Kubernetes distribution
...Compared with heavyweight PaaS platforms such as OpenShift, KubeOperator only focuses on solving one problem, which is to help enterprises plan (Day 0), deploy (Day 1), and operate (Day 2) production-level K8S clusters, and achieve the ultimate. Supports multiple computing, storage and networking scenarios. Integrate Ansible and Terraform. Support online environment and offline environment deployment. Provides a visual web UI. Supports cluster planning, deployment and operations. Easily run workloads like machine learning, high-performance computing, and more. Quickly deploy and manage applications in K8S. ... -
21
Mars Framework
Mars is a tensor-based unified framework for large-scale data
Mars is a distributed computing framework designed to scale scientific computing and data science workloads across large clusters while preserving the familiar programming interfaces of common Python libraries. The project provides a tensor-based execution model that extends the capabilities of tools such as NumPy, pandas, and scikit-learn so that large datasets can be processed in parallel without rewriting code for distributed environments. -
22
SuperEdge
An edge-native container management system for edge computing
SuperEdge is an open-source container management system for edge computing to manage compute resources and container applications in multiple edge regions. These resources and applications, in the current approach, are managed as one single Kubernetes cluster. A native Kubernetes cluster can be easily converted to a SuperEdge cluster. SuperEdge extends the powerful container orchestration and scheduling capabilities of Kubernetes to the edge. -
23
dispy
Distributed and Parallel Computing with/for Python.
dispy is a generic and comprehensive, yet easy to use framework for creating and using compute clusters to execute computations in parallel across multiple processors in a single machine (SMP), among many machines in a cluster, grid or cloud. dispy is well suited for data parallel (SIMD) paradigm where a computation (Python function or standalone program) is evaluated with different (large) datasets independently. dispy supports public / private / hybrid cloud computing, fog / edge computing. -
24
Erda
An enterprise-grade Cloud-Native application platform for Kubernetes
Erda is an open-source platform created by Terminus to ensure the development of microservice applications. It provides DevOps, microservice governance, and multi-cloud management capabilities. The multi-cloud architecture based on Kubernetes and application-centric DevOps and microservice governance can make the development, operation, monitoring, and problem diagnosis of complex business applications simpler and more efficient. erda is the core repository that implements all the RESTful... -
25
Singularity
Open source container platform designed to be simple, fast, and secure
Singularity is an open-source container platform designed to be simple, fast, and secure. Many container platforms are available, but Singularity is designed for ease of use on shared systems and in high-performance computing (HPC) environments.
