Search Results for "throughput"
Sort By:
Showing 18 open source projects for "throughput"
View related business solutions-
Earn up to 16% annual interest with Nexo.Put idle assets to work with competitive interest rates, borrow without selling, and trade with precision. All in one platform. Geographic restrictions, eligibility, and terms apply.
-
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
1
TensorRT
C++ library for high performance inference on NVIDIA GPUs
NVIDIA® TensorRT™ is an SDK for high-performance deep learning inference. It includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for deep learning inference applications. TensorRT-based applications perform up to 40X faster than CPU-only platforms during inference. With TensorRT, you can optimize neural network models trained in all major frameworks, calibrate for lower precision with high accuracy, and deploy to hyperscale data centers, embedded, or automotive product platforms. ... -
2
RTP-LLM
Alibaba's high-performance LLM inference engine for diverse apps
RTP-LLM is an open-source large language model inference acceleration engine developed by Alibaba to provide high-performance serving infrastructure for modern LLM deployments. The system focuses on improving throughput, latency, and resource utilization when running large models in production environments. It achieves this by implementing optimized GPU kernels, batching strategies, and memory management techniques tailored for transformer inference workloads. The framework is designed for large-scale AI services and is already used internally across several Alibaba platforms such as Taobao, Amap, and other business systems that rely on conversational or search-related AI services. ... -
3
FlashMLA
FlashMLA: Efficient Multi-head Latent Attention Kernels
FlashMLA is a high-performance decoding kernel library designed especially for Multi-Head Latent Attention (MLA) workloads, targeting NVIDIA Hopper GPU architectures. It provides optimized kernels for MLA decoding, including support for variable-length sequences, helping reduce latency and increase throughput in model inference systems using that attention style. The library supports both BF16 and FP16 data types, and includes a paged KV cache implementation with a block size of 64 to efficiently manage memory during decoding. On very compute-bound settings, it can reach up to ~660 TFLOPS on H800 SXM5 hardware, while in memory-bound configurations it can push memory throughput to ~3000 GB/s. ... -
4
EnvPool
C++-based high-performance parallel environment execution engine
EnvPool is a fast, asynchronous, and parallel RL environment library designed for scaling reinforcement learning experiments. Developed by SAIL at Singapore, it leverages C++ backend and Python frontend for extremely high-speed environment interaction, supporting thousands of environments running in parallel on a single machine. It's compatible with Gymnasium API and RLlib, making it suitable for scalable training pipelines. -
Train ML Models With SQL You Already KnowBuild and deploy ML models using familiar SQL. Automate data prep with built-in Gemini. Query 1 TB and store 10 GB free monthly.
-
5
bitnet.cpp
Official inference framework for 1-bit LLMs
...BitNet is built to scale across architectures, with configurable kernels and tiling strategies that adapt to different hardware, and it supports large models with impressive throughput even on modest resources. -
6
Zvec
A lightweight, lightning-fast, in-process vector database
...Developed by Alibaba’s Tongyi Lab, it positions itself as the “SQLite of vector databases” by being easy to integrate, minimal in dependencies, and capable of handling high throughput with low latency on edge devices or small systems. Zvec excels at approximate nearest neighbor search and retrieval tasks that power features like semantic search, recommendation systems, and retrieval-augmented generation (RAG) setups. Its performance benchmarks show it achieving high queries-per-second and fast index build times compared to similar tools. ... -
7
OpenMLDB
OpenMLDB is an open-source machine learning database
...However, a feature engineering script developed by data scientists (Python scripts in most cases) cannot be directly deployed into production for online inference because it usually cannot meet the engineering requirements, such as low latency, high throughput and high availability. -
8
OpenVINO Model Server
A scalable inference server for models optimized with OpenVINO
...It’s implemented in C++ for scalability and efficiency, making it suitable for both edge and cloud deployments where inference workloads must be reliable and high throughput. The server exposes model inference via standard network protocols like REST and gRPC, allowing any client that speaks those protocols to request predictions remotely, abstracting away the complexity of where and how the model runs. It supports model deployment in diverse environments including Docker, bare-metal machines, and Kubernetes clusters, and is especially useful in microservices architectures where AI services need to scale independently. ... -
9
fairseq2
FAIR Sequence Modeling Toolkit 2
fairseq2 is a modern, modular sequence modeling framework developed by Meta AI Research as a complete redesign of the original fairseq library. Built from the ground up for scalability, composability, and research flexibility, fairseq2 supports a broad range of language, speech, and multimodal content generation tasks, including instruction fine-tuning, reinforcement learning from human feedback (RLHF), and large-scale multilingual modeling. Unlike the original fairseq—which evolved into a... -
Secure File Transfer for Windows with Cerberus by RedwoodCerberus supports unlimited users and connections on a single IP, with built-in encryption, 2FA, and a browser-based web client — all deployable in under 15 minutes with a 25-day free trial.
-
10
DALI
A GPU-accelerated library containing highly optimized building blocks
...DALI addresses the problem of the CPU bottleneck by offloading data preprocessing to the GPU. Additionally, DALI relies on its own execution engine, built to maximize the throughput of the input pipeline. -
11
qvac-fabric-llm.cpp
QVAC Fabric: cross-platform LLM inference and fine-tuning
qvac-fabric-llm.cpp is a cross-platform large language model inference and fine-tuning engine built as an advanced fork of llama.cpp, designed to run efficiently across desktops, mobile devices, and heterogeneous GPU environments. The project focuses on removing hardware limitations traditionally associated with LLM deployment by enabling support for a wide range of backends, including Vulkan, Metal, CUDA, and CPU, making it accessible on devices ranging from smartphones to enterprise... -
12
Mooncake
Mooncake is the serving platform for Kimi
...Mooncake also introduces distributed key-value cache storage that allows inference systems to reuse previously computed attention states, significantly improving throughput in large-scale deployments. The system supports advanced networking technologies such as RDMA and NVMe over Fabric, enabling high-speed communication across clusters. -
13
OnnxStream
Lightweight inference library for ONNX files, written in C++
...The recommended minimum RAM/VRAM for Stable Diffusion 1.5 is typically 8GB. Generally, major machine learning frameworks and libraries are focused on minimizing inference latency and/or maximizing throughput, all of which at the cost of RAM usage. So I decided to write a super small and hackable inference library specifically focused on minimizing memory consumption: OnnxStream. OnnxStream is based on the idea of decoupling the inference engine from the component responsible for providing the model weights, which is a class derived from WeightsProvider. ... -
14
FasterTransformer
Transformer related optimization, including BERT, GPT
FasterTransformer is a high-performance inference library designed to accelerate transformer-based models such as BERT, GPT, and T5 on NVIDIA GPUs. It provides optimized implementations of transformer encoder and decoder layers using CUDA, cuBLAS, and custom kernels to maximize throughput and minimize latency. The library supports multiple deep learning frameworks, including TensorFlow, PyTorch, and Triton, allowing developers to integrate it into existing pipelines without major changes. It includes advanced optimization techniques such as mixed precision, tensor parallelism, and efficient memory management, enabling large models to run across multiple GPUs and nodes. ... -
15
LightSeq
A High Performance Library for Sequence Processing and Generation
Lightseq is a high-performance library focused on efficient inference and training for deep learning models, especially large language models (LLMs) and transformer-based architectures. Its goal is to optimize both memory usage and computational throughput, enabling faster training or inference on limited hardware while maintaining model quality. Lightseq provides optimized CUDA kernels, quantization strategies, and runtime optimizations tailored for transformer operations — which often are bottlenecks in conventional frameworks — thereby reducing memory footprint, improving speed, and making deployment of large-scale models more accessible. ... -
16
TerarkDB
A RocksDB compatible KV storage engine with better performance
...It aims to be drop-in compatible with existing RocksDB setups: you can migrate most RocksDB instances over to TerarkDB without rewriting your storage logic. Under the hood, TerarkDB employs advanced data structures and compression strategies to reduce I/O, memory usage, and latency variability — delivering higher throughput and more predictable performance under heavy load. Because of these optimizations, TerarkDB can be especially beneficial for services requiring fast read/write responses under variable workloads or those dealing with large datasets while aiming to keep resource usage efficient. Its focus includes optimizing write amplification, compaction efficiency, and storage footprint — which matters for large-scale deployment environments. -
17
TurboTransformers
Fast and user-friendly runtime for transformer inference
TurboTransformers is a high-performance inference framework optimized for running Transformer models efficiently on CPUs and GPUs. It improves latency and throughput for NLP applications. -
18
Scalable Distributed Deep-RL
A TensorFlow implementation of Scalable Distributed Deep-RL
Scalable Agent is the open implementation of IMPALA (Importance Weighted Actor-Learner Architectures), a highly scalable distributed reinforcement learning framework developed by Google DeepMind. IMPALA introduced a new paradigm for efficiently training agents across large-scale environments by decoupling acting and learning processes. In this architecture, multiple actor processes interact with their environments in parallel to collect trajectories, which are then asynchronously sent to a...
- Previous
- You're on page 1
- Next
