Search Results for "http-client"

...A planning tool calculates safe disk, RAM, and VRAM placement before loading the model, while a diagnostic command checks system readiness. Colibri includes terminal chat, an OpenAI-compatible text API, and a browser client, but disk-bound generation can be slow on cold caches.

WASTE is an embeddable C inference engine for running extremely large mixture-of-experts models when the weights exceed available RAM. It keeps the shared model trunk in memory and streams only the experts selected for each token from fast NVMe storage. A bounded cache reuses recently needed experts, while lookahead routing begins disk reads before the next layer requires them. Its main target is the full 2.78-trillion-parameter Kimi K3 model, including multimodal image input. The engine has...