Search Results for "throughput"
Sort By:
Showing 84 open source projects for "throughput"
View related business solutions-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
Build Agents and Models on One PlatformGemini Enterprise Agent Platform is Google Cloud's comprehensive platform for developers to build, scale, govern, and optimize agents and models. Choose from Google's most advanced models and third-party models like Anthropic's Claude Model Family.
-
1
vLLM
A high-throughput and memory-efficient inference and serving engine
vLLM is a fast and easy-to-use library for LLM inference and serving. High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more. -
2
DeepSpeed
Deep learning optimization library: makes distributed training easy
DeepSpeed is an easy-to-use deep learning optimization software suite that enables unprecedented scale and speed for Deep Learning Training and Inference. With DeepSpeed you can: 1. Train/Inference dense or sparse models with billions or trillions of parameters 2. Achieve excellent system throughput and efficiently scale to thousands of GPUs 3. Train/Inference on resource constrained GPU systems 4. Achieve unprecedented low latency and high throughput for inference 5. Achieve extreme compression for an unparalleled inference latency and model size reduction with low costs DeepSpeed offers a confluence of system innovations, that has made large scale DL training effective, and efficient, greatly improved ease of use, and redefined the DL training landscape in terms of scale that is possible. ... -
3
Parallax
Parallax is a distributed model serving framework
...A two-stage scheduling architecture determines how model layers are allocated to available hardware and how requests are routed across nodes during execution. This scheduling system optimizes latency, throughput, and hardware utilization even when nodes have different computational capabilities. The platform also supports model sharding and pipeline parallelism, allowing very large models to run across distributed resources. -
4
LitServe
Minimal Python framework for scalable AI inference servers fast
...Unlike traditional serving tools that enforce rigid abstractions, LitServe focuses on flexibility by letting users control request handling, batching strategies, and output processing directly in Python. LitServe is built on top of FastAPI and extends it with AI-specific optimizations such as efficient multi-worker execution, which can significantly improve throughput. It includes built-in capabilities for batching, streaming responses, and automatic scaling across CPUs and GPUs, enabling high-performance deployments. -
Veeam Data Platform v13.1 - Get Your Free TrialTry Veeam Data Platform today. Experience the unified platform that's secure by design, portable by default, and proven to recover clean, fast, and anywhere.
-
5
Text Embeddings Inference
High-performance inference server for text embeddings models API layer
...It provides an API interface that allows developers to integrate embedding capabilities into applications without managing model internals directly. Text Embeddings Inference is optimized for throughput and low latency, enabling it to handle large volumes of requests reliably. It also emphasizes ease of deployment, often using containerization and configurable runtime options to adapt to different infrastructure setups. -
6
FlexLLMGen
Running large language models on a single GPU
...This design allows organizations to deploy powerful language models for high-volume tasks without the infrastructure costs typically associated with large-scale AI systems. The project is particularly useful for workloads that prioritize throughput over latency, including benchmarking experiments and large corpus analysis. -
7
MemOS
AI memory OS for LLM and Agent systems
...By abandoning some of the historical assumptions of Unix-style operating systems, MemOS attempts to unlock new performance and scalability tradeoffs for applications that need high throughput and low latency on memory-intensive workloads. -
8
Infinity
Low-latency REST API for serving text-embeddings
Infinity is a high-throughput, low-latency REST API for serving vector embeddings, supporting all sentence-transformer models and frameworks. Infinity is developed under MIT License. Infinity powers inference behind Gradient.ai and other Embedding API providers. -
9
DeepSpeed MII
MII makes low-latency and high-throughput inference possible
MII makes low-latency and high-throughput inference possible, powered by DeepSpeed. The Deep Learning (DL) open-source community has seen tremendous growth in the last few months. Incredibly powerful text generation models such as the Bloom 176B, or image generation model such as Stable Diffusion are now available to anyone with access to a handful or even a single GPU through platforms such as Hugging Face. -
Custom VMs From 1 to 96 vCPUs With 99.95% UptimeLive migration and automatic failover keep workloads online through maintenance. One free e2-micro VM every month.
-
10
JoyAI-Video-Edit
Real-Time Open-Ended Video Editing with Autoregressive Diffusion
...Its architecture combines a multimodal condition encoder, causal video VAE, and a 16B-parameter multimodal diffusion transformer. Autoregressive diffusion and bounded KV-state inference are used to keep computation stable across long streams. The released deployment reaches high-throughput 720p editing and also supports real-time operation on selected consumer GPUs. Updated checkpoints improve identity preservation, reference conditioning, and temporal consistency for reference-image-guided editing. -
11
alive-progress
A new kind of Progress Bar, with real-time throughput, ETA
alive-progress is an advanced Python progress bar library that introduces a highly animated and adaptive approach to tracking long-running tasks. Unlike traditional static progress indicators, it dynamically adjusts spinner speed and visual feedback based on actual throughput, giving users a more intuitive sense of activity. The library is designed with performance efficiency in mind, using multithreaded updates that minimize CPU overhead and terminal noise. It includes sophisticated ETA estimation powered by exponential smoothing algorithms, improving prediction accuracy for variable workloads. Developers can easily integrate it into scripts thanks to automatic logging hooks and flexible configuration options. ... -
12
SimpleLLM
950 line, minimal, extensible LLM inference engine built from scratch
...Designed to run efficiently on high-end GPUs like NVIDIA H100 with support for models such as OpenAI/gpt-oss-120b, Simple-LLM implements continuous batching and event-driven inference loops to maximize hardware utilization and throughput. Its straightforward code structure allows anyone experimenting with custom kernels, new batching strategies, or inference optimizations to trace execution from input to output with minimal cognitive overhead. -
13
Text Generation Inference
Large Language Model Text Generation Inference
Text Generation Inference is a high-performance inference server for text generation models, optimized for Hugging Face's Transformers. It is designed to serve large language models efficiently with optimizations for performance and scalability. -
14
iptv-api
IPTV live stream source automatic update tool
...Users can customize channel templates, aliases, logos, EPG data, protocol preferences, geographic filters, providers, resolution, and speed requirements. The system measures latency, throughput, resolution, and frame rate while removing invalid, unavailable, or repetitive placeholder streams. Results can be categorized, cached, logged, analyzed, frozen, and restored as source quality changes. Deployment options include GitHub Actions workflows, a command-line interface, a graphical interface, and Docker images for several processor architectures. -
15
Motor
The async Python driver for MongoDB and Tornado or asyncio
...It provides a familiar API surface similar to the official synchronous PyMongo driver, so you can migrate or write MongoDB code in Python without having to learn a completely new interface. Because it integrates with popular async ecosystems like FastAPI, Sanic, and aiohttp, Motor is a natural fit for modern async Python stacks where throughput and responsiveness matter. It also supports change streams, grid file system (GridFS), and the full range of CRUD and aggregation operations available in MongoDB. -
16
CoreNet
CoreNet: A library for training deep neural networks
...CoreNet provides abstractions for data, tensor, and pipeline parallelism, allowing models to scale without code duplication or heavy manual configuration. Its distributed runtime manages synchronization, load balancing, and mixed-precision computation to maximize throughput while minimizing communication bottlenecks. CoreNet integrates tightly with Apple’s proprietary ML stack and hardware, serving as the foundation for research in computer vision, language models, and multimodal systems within Apple AI. The framework includes monitoring tools, fault tolerance mechanisms, and efficient checkpointing for massive training runs. -
17
FlashAttention
Fast and memory-efficient exact attention
...The project provides implementations of FlashAttention, FlashAttention-2, and newer iterations optimized for modern GPU architectures such as NVIDIA Hopper and AMD accelerators. By improving both forward and backward pass efficiency, it enables training and inference of large language models with longer sequence lengths and higher throughput. The library integrates with PyTorch and supports various attention configurations, including causal masking, multi-query attention, and rotary embeddings. -
18
LMCache
Supercharge Your LLM with the Fastest KV Cache Layer
...Instead of rebuilding KV states for repeated or shared text segments, LMCache persists and retrieves them from multiple tiers—GPU memory, CPU DRAM, and local disk—then injects them into subsequent requests to reduce TTFT and increase throughput. Its design supports reuse beyond strict prefix matching and enables sharing across serving instances, improving efficiency under real multi-tenant traffic. The broader project includes examples, tests, a server component, and public posts describing cross-engine sharing and inter-GPU KV transfers. These capabilities aim to lower latency, cut GPU cycles, and stabilize performance for production workloads with overlapping prompts or retrieval-augmented contexts. ... -
19
DeepEP
DeepEP: an efficient expert-parallel communication library
DeepEP is a communication library designed specifically to support Mixture-of-Experts (MoE) and expert parallelism (EP) deployments. Its core role is to implement high-throughput, low-latency all-to-all GPU communication kernels, which handle the dispatching of tokens to different experts (or shards) and then combining expert outputs back into the main data flow. Because MoE architectures require routing inputs to different experts, communication overhead can become a bottleneck — DeepEP addresses that by providing optimized GPU kernels and efficient dispatch/combining logic. ... -
20
FramePack
Lets make video diffusion practical
FramePack explores compact representations for sequences of image frames, targeting tasks where many near-duplicate frames carry redundant information. The idea is to “pack” frames by detecting shared structure and storing differences efficiently, which can accelerate training or inference on video-like data. By reducing I/O and memory bandwidth, datasets become lighter to load while models still see the essential temporal variation. The repository demonstrates both packing and unpacking... -
21
HypoMux
Windows multi-NIC bandwidth aggregator
...It is designed for PCs connected to multiple independent networks, such as Ethernet, Wi-Fi, and mobile hotspot connections. The app distributes different download connections across available network interfaces so multi-threaded downloads can use more combined throughput. Version 2 adds a virtual network adapter mode alongside the original system proxy mode for broader traffic handling. It also includes process-level routing rules so latency-sensitive apps can bypass aggregation while download-heavy apps use it. HypoMux is best suited for large Steam updates, download managers, browser downloads, and similar multi-connection workloads, not single-stream transfers. -
22
Lemonade
Lemonade helps users run local LLMs with the highest performance
...The repository highlights easy onboarding with downloads, docs, and a Discord for support, suggesting an active user community. Messaging centers on squeezing maximum throughput/latency from modern accelerators without users having to hand-tune kernels or flags. Releases further reinforce the “server” framing, pointing developers toward a service that can be integrated into apps and tools. -
23
TensorRT LLM
TensorRT LLM provides users with an easy-to-use Python API
...It provides a Python-based API built on top of PyTorch that allows developers to define, customize, and deploy LLMs efficiently across a variety of hardware configurations, from single GPUs to large multi-node clusters. The library focuses on maximizing throughput and minimizing latency through advanced techniques such as quantization, custom attention kernels, and optimized memory management strategies. It includes support for cutting-edge inference methods like speculative decoding and inflight batching, enabling real-time and large-scale AI applications. TensorRT-LLM integrates seamlessly with NVIDIA’s broader inference ecosystem, including Triton Inference Server and distributed deployment frameworks, making it suitable for production environments. -
24
slime LLM
slime is an LLM post-training framework for RL Scaling
slime is an open-source large language model (LLM) post-training framework developed to support reinforcement learning (RL)-based scaling and high-performance training workflows for advanced LLMs, blending training and rollout modules into an extensible system. It offers a flexible architecture that connects high-throughput training (e.g., via Megatron-LM) with a customizable data generation pipeline, enabling researchers and engineers to iterate on new RL training paradigms effectively. The framework is designed to support a wide range of training modes, allowing both synchronous and asynchronous RL workflows and programmable rollout interfaces that simplify experimentation with custom environments and reward signals. ... -
25
Scweet
Scrape tweets, profiles, followers and following from Twitter/X
Scweet is a Python-based Twitter/X scraping library and CLI designed to collect tweets, profile timelines, followers, following lists, and user profile data without requiring the official Twitter/X API or a developer account. Instead of depending on deprecated unauthenticated scraping methods, it works by using X’s web GraphQL API together with authenticated browser cookies, which gives it a more current and practical approach for data extraction. The project supports a broad set of...
