Search Results for "inference engine"

uzu is a high-performance inference engine designed to run artificial intelligence models efficiently on Apple Silicon hardware. Written primarily in Rust and leveraging Apple’s Metal framework, the project focuses on maximizing performance when executing large language models and other AI workloads on devices such as Mac computers with M-series chips. The engine implements a hybrid architecture in which model layers can be executed either as custom GPU kernels or through Apple’s MPSGraph API, allowing it to balance performance and compatibility depending on the workload. ...

...Its server implements OpenAI-compatible chat, completion, embedding, transcription, tool-calling, and multimodal endpoints. The custom .base format supports affine quantization from Q2 through Q8, optional AWQ calibration, and signed model bundles. Stable C interfaces connect the engine with Python, Node.js, Rust, and Swift applications. The repository contains the open CLI, format specifications, bindings, documentation, and benchmarks, while the prebuilt inference engine uses a separate license.

mistral.rs is a fast and flexible LLM inference engine implemented in Rust, designed to run and serve modern language models with an emphasis on performance and practical deployment. It provides multiple entry points for developers, including a CLI for running models locally and an HTTP server that exposes an OpenAI-compatible API surface for easy integration with existing clients.

...It ships as a roughly 15 MB binary with no external runtime dependencies. Users can work through an interactive terminal interface, headless commands, a server protocol, or Slack and Discord gateways. A built-in inference engine can run GGUF models entirely offline without an account, API key, or internet connection. Ante also supports more than a dozen hosted providers and can switch between commercial, open-weight, and local models. Multi-agent orchestration lets it spawn and coordinate specialized subagents for larger software tasks. Skills, MCP integrations, persistent memory, resumable sessions, and public benchmark evaluation extend it into a lightweight general-purpose agent harness.