Search Results for "load balancing"
Sort By:
Showing 21 open source projects for "load balancing"
View related business solutions-
Build Data Resilience - Take the Assessment TodayIs your recovery strategy as strong as you think? Take this quick self-assessment to check your recovery readiness and gain tailored insights. In only 2 minutes, you'll learn where you fall on the recovery readiness scale.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
AxonHub
Use any SDK to call 100+ LLMs
...AxonHub translates requests from one provider’s API format into another, enabling seamless interoperability across different AI platforms. The system also provides infrastructure features such as request routing, failover mechanisms, load balancing, and cost management for AI applications. This architecture makes it easier to experiment with multiple models and manage production deployments that rely on several providers simultaneously. -
2
ClawRouter
Smart LLM router
ClawRouter is a flexible networking and routing framework designed to support AI-oriented distributed systems and agent ecosystems by managing how messages, requests, and responses are routed between components. It provides a programmable router abstraction that can handle complex traffic patterns, enabling dynamic message forwarding, load balancing, and custom routing logic based on content, context, or policy rules. Because distributed AI systems often involve many services, agents, and runtime components interacting with each other and with external APIs, ClawRouter helps ensure that communication paths remain clear, efficient, and adaptable as systems scale. The framework supports plugin-based extensions so developers can define custom protocols, transformation hooks, and monitoring handlers without modifying core routing logic. ... -
3
DeepSeek-V3
Powerful AI language model (MoE) optimized for efficiency/performance
DeepSeek-V3 is a robust Mixture-of-Experts (MoE) language model developed by DeepSeek, featuring a total of 671 billion parameters, with 37 billion activated per token. It employs Multi-head Latent Attention (MLA) and the DeepSeekMoE architecture to enhance computational efficiency. The model introduces an auxiliary-loss-free load balancing strategy and a multi-token prediction training objective to boost performance. Trained on 14.8 trillion diverse, high-quality tokens, DeepSeek-V3 underwent supervised fine-tuning and reinforcement learning to fully realize its capabilities. Evaluations indicate that it outperforms other open-source models and rivals leading closed-source models, achieving this with a training duration of 55 days on 2,048 Nvidia H800 GPUs, costing approximately $5.58 million. -
4
Kong
The Cloud-Native API Gateway
...When building for the web, mobile, or Internet of Things, you’ll need a common functionality to run your software, and Kong is that solution. Kong acts as a gateway, connecting microservices requests and APIs natively while also providing load balancing, logging, monitoring, authentication, rate-limiting, and so much more through plugins. Kong is highly extensible as well as platform agnostic, connecting APIs across different environments, platforms and patterns. Achieve architectural freedom with Kong today. -
Ship Agents FasterGemini Enterprise Agent Platform lets you rapidly build, scale, govern and optimize production-ready agents grounded in your organization's data. The platform enables developers to build custom or pre-built agents for virtually any use case. New customers get $300 in free credits.
-
5
Paddler
Open-source LLM load balancer and serving platform for hosting LLMs
Paddler is an open-source LLM infrastructure platform designed to deploy, manage, and scale large language models on private infrastructure. The system acts as a specialized load balancer and serving layer for language models, enabling organizations to run inference workloads without relying on external API providers. It supports running models locally through engines such as llama.cpp while distributing requests across multiple compute nodes to improve performance and reliability. The... -
6
System Design Academy
If you want to become good at AI & system design, join this newsletter
...Real-world examples examine how major services scale storage, messaging, streaming, payments, collaboration, and other workloads. Fundamental topics include APIs, caching, consistency, databases, DNS, Docker, Kafka, Kubernetes, load balancing, microservices, and service discovery. Separate guides cover interview frameworks and design exercises for products such as ChatGPT, Spotify, WhatsApp, and YouTube. The repository also collects AI agent concepts and selected software architecture white papers. -
7
LangDB AI Gateway
Govern, secure, and optimize your AI traffic
AI Gateway is a high-performance, open-source API gateway optimized for managing and monitoring LLM traffic at scale. Developed by the LangDB team, AI Gateway acts as an intermediary between clients and backend LLMs, providing advanced features like caching, rate limiting, prompt management, and observability. It helps teams secure and optimize their LLM deployments, whether using local models or external APIs like OpenAI or Anthropic. With native support for multi-tenant environments and... -
8
Pruna AI
Pruna is a model optimization framework built for developers
Pruna is an open-source, self-hostable AI inference engine designed to help teams deploy and manage large language models (LLMs) efficiently across private or hybrid infrastructures. Built with performance and developer ergonomics in mind, Pruna simplifies inference workflows by enabling multi-model orchestration, autoscaling, GPU resource allocation, and compatibility with popular open-source models. It is ideal for companies or teams looking to reduce reliance on external APIs while... -
9
One API
The LLM API management & distribution system
One API is an open-source platform designed to unify access to multiple AI model providers through a single, consistent API interface. It acts as a gateway that allows developers to manage and route requests to different large language models and AI services without needing to integrate each provider separately. The system supports multiple backends, enabling users to switch between providers or balance usage based on cost, performance, or availability. It includes features such as... -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
10
Swarms
Enterprise multi-agent orchestration framework for scalable AI apps
Swarms is an enterprise-grade multi-agent orchestration framework designed to help developers build, manage, and scale collaborative AI systems composed of multiple agents. It provides a structured infrastructure for coordinating agents in hierarchical, parallel, or sequential workflows, enabling complex task execution across distributed components. It emphasizes production readiness, offering modular architecture, high availability, and observability features suitable for large-scale... -
11
Parallax
Parallax is a distributed model serving framework
Parallax is a decentralized inference framework designed to run large language models across distributed computing resources. Instead of relying on centralized GPU clusters in data centers, the system allows multiple heterogeneous machines to collaborate in serving AI inference workloads. Parallax divides model layers across different nodes and dynamically coordinates them to form a complete inference pipeline. A two-stage scheduling architecture determines how model layers are allocated to... -
12
Bifrost
The Fastest LLM Gateway with built in OTel observability
...It is built to be high performance: in benchmark tests at 5,000 requests per second, it reportedly adds only microseconds of overhead and achieves perfect success rates with no failed requests. Bifrost supports features such as automatic fallback (failover between providers), load balancing across API keys/providers, and semantic caching to reduce latency and cost. It also includes observability with built-in metrics, tracing, logging, and supports governance features like rate limiting, access control, and cost budgeting. The architecture is modular: there is a core engine, plugin layers, and transport layers (HTTP APIs). -
13
Portkey AI Gateway
A blazing fast AI Gateway with integrated guardrails
...It presents a single, friendly API through which you can route to 200+ LLMs, while applying configurable input/output guardrails to enforce policies or restrict certain content. It supports automatic retries, fallbacks, load balancing across providers or keys, and request timeouts to avoid latency spikes. The gateway is multimodal: it can handle text, vision, audio, and image models under a common interface. It also offers features for governance: role-based access, compliance with standards (SOC2, HIPAA, GDPR), secure key management, and logging/analytics of usage, latency, errors, and cost. ... -
14
Defang
Defang CLI and sample projects
...The platform supports multiple programming languages, including Go, JavaScript, and Python, allowing developers to start with sample projects or generate project outlines using natural language prompts. With a single command, Defang builds and deploys applications, handling configurations for computing, storage, load balancing, networking, logging, and security. The Defang Command Line Interface (CLI) facilitates interactions with the platform, offering installation options via shell scripts, Homebrew, Winget, Nix, or direct download. Developers can define services using compose.yaml files, which Defang utilizes to deploy applications to the cloud. -
15
Envoy AI Gateway
Manages Unified Access to Generative AI Services
Envoy AI Gateway is an open-source gateway system designed to manage network traffic between applications and generative AI services using the Envoy proxy ecosystem. The project extends Envoy Gateway to support AI-specific workloads, enabling organizations to route, secure, and scale requests to large language models and other generative AI services. In a typical deployment, the architecture uses a two-tier gateway model where an outer gateway handles authentication, routing, and global rate... -
16
APIPark
APIPark is the #1 open-source AI Gateway and Developer Portal
...For example, using OpenAI GPT-4 and custom prompts, you can create sentiment analysis APIs, translation APIs, or data analysis APIs. API lifecycle management helps standardize the process of managing APIs, including traffic forwarding, load balancing, and managing different versions of publicly accessible APIs. This improves API quality and maintainability. -
17
GoModel
The best AI Gateway 2026 - GoModel
GoModel is the last AI Gateway you'll ever try. It's the most secure, the fastest, and the most reliable AI Gateway according to self-verifiable benchmarks. The best open-source alternative to LiteLLM, GoModel provides a unified OpenAI-compatible API for OpenAI, Anthropic, Gemini, Groq, xAI, Ollama, vLLM, Amazon Bedrock, OpenRouter, and many more providers. Built in Go, it's lightweight, resource-efficient, and designed for production workloads with extremely low latency. GoModel... -
18
GPTLink
Build a free commercial AI dialogue environment in 10 minutes
GPTLink is an open-source platform designed to manage, proxy, and distribute access to large language model APIs, enabling organizations and developers to centralize AI usage and optimize resource allocation. It acts as an intermediary layer between users and multiple AI providers, allowing requests to be routed, balanced, and monitored through a unified interface. The system supports API key management, usage tracking, and rate limiting, making it particularly useful for teams that need to... -
19
CLIP-as-service
Embed images and sentences into fixed-length vectors
...Non-blocking duplex streaming on requests and responses, designed for large data and long-running tasks. Horizontally scale up and down multiple CLIP models on single GPU, with automatic load balancing. Easy-to-use. No learning curve, minimalist design on client and server. Intuitive and consistent API for image and sentence embedding. Async client support. Easily switch between gRPC, HTTP, WebSocket protocols with TLS and compression. Smooth integration with neural search ecosystem including Jina and DocArray. Build cross-modal and multi-modal solutions in no time. -
20
FastoCloud PRO
IPTV/NVR/CCTV/Video cloud https://fastocloud.com
IPTV/Video cloud Features: Cross-platform (Linux, MacOSX, FreeBSD, Raspbian/Armbian) GPU/CPU Encode/Decode/Post Processing Stream statistics CCTV Adaptive hls streams Load balancing Temporary urls HLS push EPG scanning Subtitles to text conversions AD insertion Logo overlay Video effects Relays Timeshifts Catchups Playlists Restream/Transcode from online streaming services like Youtube, Twitch Mozaic Many Outputs Physical Inputs Streaming Protocols File Formats Presets Vods/Series server-side support Pay per view channels Channels on demand HTTP Live Streaming (HLS) server-side support Public API, client server communication via JSON RPC Protocol gzip compression Deep learning video analysis Supported deep learning frameworks: Tensorflow NCSDK Caffe ML Hardware: -
21
Agent.GUI
The Project moved to github https://github.com/EnFlexIT/AgentWorkbench
The project has moved to github https://github.com/EnFlexIT/AgentWorkbench Agent.GUI is a simulation framework and toolkit based on the JADE framework. It provides functionalities for time aspects, agent-environment interaction, visualization and load balancing, Furthermore, the included application focuses the usability for end users.
- Previous
- You're on page 1
- Next