Browse free open source C++ LLM Inference Tools and projects below. Use the toggles on the left to filter open source C++ LLM Inference Tools by OS, license, language, programming language, and project status.
Port of OpenAI's Whisper model in C/C++
Port of Facebook's LLaMA model in C/C++
Run Local LLMs on Any Device. Open-source
OpenVINO™ Toolkit repository
ONNX Runtime: cross-platform, high performance ML inferencing
C++ library for high performance inference on NVIDIA GPUs
MNN is a blazing fast, lightweight deep learning framework
C++ implementation of ChatGLM-6B & ChatGLM2-6B & ChatGLM3 & GLM4(V)
Easy-to-use deep learning framework with 3 key features
High-performance neural network inference framework for mobile
Fast inference engine for Transformer models
Connect home devices into a powerful cluster to accelerate LLM
LLMs as Copilots for Theorem Proving in Lean
Lightweight inference library for ONNX files, written in C++
The deep learning toolkit for speech-to-text
Deep Learning API and Server in C++14 support for Caffe, PyTorch
Open standard for machine learning interoperability
PArallel Distributed Deep LEarning: Machine Learning Framework
Easy-to-use Speech Toolkit including Self-Supervised Learning model
Bolt is a deep learning library with high performance
Pure C++ implementation of several models for real-time chatting
Deep learning inference framework optimized for mobile platforms
Set of comprehensive computer vision & machine intelligence libraries
lightweight, standalone C++ inference engine for Google's Gemma models
On-device AI across mobile, embedded and edge for PyTorch