Search Results for "inference engine expert"
Sort By:
Showing 166 open source projects for "inference engine expert"
View related business solutions-
Build Data Resilience - Take the Assessment TodayIs your recovery strategy as strong as you think? Take this quick self-assessment to check your recovery readiness and gain tailored insights. In only 2 minutes, you'll learn where you fall on the recovery readiness scale.
-
Earn up to 16% annual interest with Nexo.Generate interest, borrow against your crypto, and trade a range of cryptocurrencies — all in one platform. Geographic restrictions, eligibility, and terms apply.
-
1
Transformer Engine
A library for accelerating Transformer models on NVIDIA GPUs
Transformer Engine (TE) is a library for accelerating Transformer models on NVIDIA GPUs, including using 8-bit floating point (FP8) precision on Hopper GPUs, to provide better performance with lower memory utilization in both training and inference. TE provides a collection of highly optimized building blocks for popular Transformer architectures and an automatic mixed precision-like API that can be used seamlessly with your framework-specific code. -
2
Temporal Inference Engine
A real time inference engine for temporal logical specifications
A real time inference engine for temporal logical specifications, which is able to acquire, process and generate any binary or real signal through POSIX IPC, files or UNIX sockets. Specifications of signals and dynamic systems are represented as special graphs and executed in real time, with a predictable sampling time of few milliseconds. Real time signal processing, dynamic system control, state machine modeling and logical property verification are some fields of application of this software. ... -
3
MLX Engine
LM Studio Apple MLX engine
MLX Engine is the Apple MLX-based inference backend used by LM Studio to run large language models efficiently on Apple Silicon hardware. Built on top of the mlx-lm and mlx-vlm ecosystems, the engine provides a unified architecture capable of supporting both text-only and multimodal models. Its design focuses on high-performance on-device inference, leveraging Apple’s MLX stack to accelerate computation on M-series chips. -
4
Kimi K3 in C
A 2.78-trillion-parameter Kimi K3 running inference on a single CPU
Kimi K3 in C is a portable C99 inference engine built to run the 2.78-trillion-parameter Kimi K3 model on CPUs without BLAS, machine-learning frameworks, or GPUs. It demonstrates inference from a roughly 1.56 TB checkpoint with measured memory use as low as 8.24 GB. The runtime streams model trunk layers and routed experts from disk instead of keeping all weights resident in memory. -
MongoDB Atlas runs apps anywhereMongoDB Atlas gives you the freedom to build and run modern applications anywhere—across AWS, Azure, and Google Cloud. With global availability in over 115 regions, Atlas lets you deploy close to your users, meet compliance needs, and scale with confidence across any geography.
-
5
WASTE
Run the full 2.78-trillion-parameter Kimi K3 model
WASTE is an embeddable C inference engine for running extremely large mixture-of-experts models when the weights exceed available RAM. It keeps the shared model trunk in memory and streams only the experts selected for each token from fast NVMe storage. A bounded cache reuses recently needed experts, while lookahead routing begins disk reads before the next layer requires them. -
6
Colibrì
Run frontier MoE models on hardware you already own
Colibrì is an open-source inference engine designed to run frontier-scale mixture-of-experts (MoE) models on consumer and heterogeneous hardware. It treats VRAM, system RAM, and NVMe storage as a unified memory hierarchy, dynamically placing and streaming model weights where they can be accessed most efficiently. This architecture enables users to run models ranging from 7 billion to 2.8 trillion parameters without requiring the entire model to fit in expensive GPU memory. ... -
7
DeepEP
DeepEP: an efficient expert-parallel communication library
...The library also supports low-precision operations (such as FP8) to reduce memory and bandwidth usage during communication. DeepEP is aimed at large-scale model inference or training systems where expert parallelism is used to scale model capacity without replicating entire networks. -
8
ReactiveMP.jl
High-performance reactive message-passing based Bayesian engine
ReactiveMP.jl is a Julia package that provides an efficient reactive message passing based Bayesian inference engine on a factor graph. The package is a part of the bigger and user-friendly ecosystem for automatic Bayesian inference called RxInfer. While ReactiveMP.jl exports only the inference engine, RxInfer provides convenient tools for model and inference constraints specification as well as routines for running efficient inference both for static and real-time datasets. -
9
Ling
Ling is a MoE LLM provided and open-sourced by InclusionAI
Ling is a Mixture-of-Experts (MoE) large language model (LLM) provided and open-sourced by inclusionAI. The project offers different sizes (Ling-lite, Ling-plus) and emphasizes flexibility and efficiency: being able to scale, adapt expert activation, and perform across a range of natural language/reasoning tasks. Example scripts, inference pipelines, and documentation. The codebase includes inference, examples, models, documentation, and model download infrastructure. As more developers and researchers engage with the platform, we can expect rapid advancements and improvements, leading to even more sophisticated applications. ... -
AI-generated apps that pass security reviewRetool lets you generate dashboards, admin panels, and workflows directly on your data. Type something like “Build me a revenue dashboard on my Stripe data” and get a working app with security, permissions, and compliance built in from day one. Whether on our cloud or self-hosted, create the internal software your team needs without compromising enterprise standards or control.
-
10
ds4.c
DeepSeek 4 Flash local inference engine for Metal
ds4.c is a specialized local inference engine created by antirez for running DeepSeek V4 Flash models directly on Apple Silicon hardware using Metal acceleration. Unlike general-purpose inference runtimes, the project is intentionally optimized for a specific model family, enabling highly efficient execution and simplified architecture. The engine includes DS4-specific model loading, KV cache management, prompt rendering, and OpenAI-compatible server APIs for local deployment workflows. ... -
11
vLLM
A high-throughput and memory-efficient inference and serving engine
vLLM is a fast and easy-to-use library for LLM inference and serving. High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more. -
12
Tencent-Hunyuan-Large
Open-source large language model family from Tencent Hunyuan
Tencent-Hunyuan-Large is the flagship open-source large language model family from Tencent Hunyuan, offering both pre-trained and instruct (fine-tuned) variants. It is designed with long-context capabilities, quantization support, and high performance on benchmarks across general reasoning, mathematics, language understanding, and Chinese / multilingual tasks. It aims to provide competitive capability with efficient deployment and inference. FP8 quantization support to reduce memory usage... -
13
Nano-vLLM
A lightweight vLLM implementation built from scratch
Nano-vLLM is a lightweight implementation of the vLLM inference engine designed to run large language models efficiently while maintaining a minimal and readable codebase. The project recreates the core functionality of vLLM in a simplified architecture written in approximately a thousand lines of Python, making it easier for developers and researchers to understand how modern LLM inference systems work. -
14
Ring
Ring is a reasoning MoE LLM provided and open-sourced by InclusionAI
Ring is a reasoning Mixture-of-Experts (MoE) large language model (LLM) developed by inclusionAI. It is built from or derived from Ling. Its design emphasizes reasoning, efficiency, and modular expert activation. In its “flash” variant (Ring-flash-2.0), it optimizes inference by activating only a subset of experts. It applies reinforcement learning/reasoning optimization techniques. Its architectures and training approaches are tuned to enable efficient and capable reasoning performance. Reasoning-optimized model with reinforcement learning enhancements. ... -
15
SimpleLLM
950 line, minimal, extensible LLM inference engine built from scratch
SimpleLLM is a minimal, extensible large language model inference engine implemented in roughly 950 lines of code, built from scratch to serve both as a learning tool and a research platform for novel inference techniques. It provides the core components of an LLM runtime—such as tokenization, batching, and asynchronous execution—without the abstraction overhead of more complex engines, making it easier for developers and researchers to understand and modify. ... -
16
uzu
A high-performance inference engine for AI models
uzu is a high-performance inference engine designed to run artificial intelligence models efficiently on Apple Silicon hardware. Written primarily in Rust and leveraging Apple’s Metal framework, the project focuses on maximizing performance when executing large language models and other AI workloads on devices such as Mac computers with M-series chips. The engine implements a hybrid architecture in which model layers can be executed either as custom GPU kernels or through Apple’s MPSGraph API, allowing it to balance performance and compatibility depending on the workload. ... -
17
DeepSpeed
Deep learning optimization library: makes distributed training easy
DeepSpeed is an easy-to-use deep learning optimization software suite that enables unprecedented scale and speed for Deep Learning Training and Inference. With DeepSpeed you can: 1. Train/Inference dense or sparse models with billions or trillions of parameters 2. Achieve excellent system throughput and efficiently scale to thousands of GPUs 3. Train/Inference on resource constrained GPU systems 4. Achieve unprecedented low latency and high throughput for inference 5. Achieve extreme... -
18
gemma.cpp
lightweight, standalone C++ inference engine for Google's Gemma models
Gemma.cpp is a C++ implementation for running inference with Gemma models efficiently on CPUs and GPUs. Developed by Google, it allows running large language models (LLMs) like Gemma with minimal hardware, focusing on optimized performance and low latency. Gemma.cpp is intended for developers seeking to deploy LLMs in production environments without needing massive computational resources. -
19
Jlama
Jlama is a modern LLM inference engine for Java
Jlama is a modern inference engine written entirely in Java that enables developers to run large language models locally within Java applications. Unlike frameworks that require external APIs or remote services, Jlama performs inference directly on a machine using pre-trained models. This allows organizations to integrate generative AI features into their systems while maintaining full control over data privacy and infrastructure. -
20
Hasura GraphQL Engine
Blazing fast, instant realtime GraphQL APIs on your DB
...Run Hasura, locally or in the cloud, and connect it to your new or existing databases to instantly get a production-grade GraphQL API. Developers and architects love Hasura because it takes no time to get started, doesn’t need them to be a GraphQL expert upfront, and saves their teams months of recurring effort in building, shipping, and maintaining their APIs. Hasura’s built-in RLS style authorization engine allows you to conveniently specify authorization rules at a model level, and safely expose the GraphQL API to developers inside or outside your organization. Hasura’s authz engine is enabling agile teams in fast-growing startups as well as powering mission-critical data access in highly regulated environments such as Fortune 500 healthcare, financial services and US federal agencies. -
21
TokenSpeed
TokenSpeed is a speed-of-light LLM inference engine
TokenSpeed is an LLM inference engine designed for high-performance production agent workloads. It aims to combine TensorRT-LLM-level speed with vLLM-level usability, making it relevant for teams that need fast generation without sacrificing developer ergonomics. The project is focused on the specific needs of agentic systems, where latency, throughput, and efficient scheduling matter across many short or tool-heavy requests. -
22
Chitu
High-performance inference framework for large language models
Chitu is a high-performance inference engine designed to deploy and run large language models efficiently in production environments. The framework focuses on improving efficiency, flexibility, and scalability for organizations that need to run LLM inference workloads across different hardware platforms. It supports heterogeneous computing environments, including CPUs, GPUs, and various specialized AI accelerators, allowing models to run across a wide range of infrastructure configurations. ... -
23
qvac-fabric-llm.cpp
QVAC Fabric: cross-platform LLM inference and fine-tuning
qvac-fabric-llm.cpp is a cross-platform large language model inference and fine-tuning engine built as an advanced fork of llama.cpp, designed to run efficiently across desktops, mobile devices, and heterogeneous GPU environments. The project focuses on removing hardware limitations traditionally associated with LLM deployment by enabling support for a wide range of backends, including Vulkan, Metal, CUDA, and CPU, making it accessible on devices ranging from smartphones to enterprise servers. ... -
24
llama.cpp
LLM inference in C/C++
llama.cpp is a high-performance C and C++ project for running large language models locally and in the cloud with minimal setup. It is built around efficient inference, broad hardware support, and the GGUF model format. The project supports many model families and has become a major foundation for local AI tools, model serving, and embedded inference workflows. It provides command-line tools, a server mode with an OpenAI-compatible API style, model conversion utilities, and extensive backend... -
25
RTP-LLM
Alibaba's high-performance LLM inference engine for diverse apps
RTP-LLM is an open-source large language model inference acceleration engine developed by Alibaba to provide high-performance serving infrastructure for modern LLM deployments. The system focuses on improving throughput, latency, and resource utilization when running large models in production environments. It achieves this by implementing optimized GPU kernels, batching strategies, and memory management techniques tailored for transformer inference workloads. ...
