Search Results for "nvme"

WASTE is an embeddable C inference engine for running extremely large mixture-of-experts models when the weights exceed available RAM. It keeps the shared model trunk in memory and streams only the experts selected for each token from fast NVMe storage. A bounded cache reuses recently needed experts, while lookahead routing begins disk reads before the next layer requires them. Its main target is the full 2.78-trillion-parameter Kimi K3 model, including multimodal image input. The engine has no third-party runtime dependency on its CPU inference path and exposes both a CLI and C library. ...

Colibrì is an open-source inference engine designed to run frontier-scale mixture-of-experts (MoE) models on consumer and heterogeneous hardware. It treats VRAM, system RAM, and NVMe storage as a unified memory hierarchy, dynamically placing and streaming model weights where they can be accessed most efficiently. This architecture enables users to run models ranging from 7 billion to 2.8 trillion parameters without requiring the entire model to fit in expensive GPU memory. Written primarily in pure C, Colibrì has no core engine dependencies and can operate without a GPU, although CUDA, Metal, Vulkan, and other acceleration options can improve performance. ...

...Mooncake also introduces distributed key-value cache storage that allows inference systems to reuse previously computed attention states, significantly improving throughput in large-scale deployments. The system supports advanced networking technologies such as RDMA and NVMe over Fabric, enabling high-speed communication across clusters.