Browse free open source LLM Inference tools and projects for Linux below. Use the toggles on the left to filter open source LLM Inference tools by OS, license, language, programming language, and project status.
Port of OpenAI's Whisper model in C/C++
Run Local LLMs on Any Device. Open-source
ONNX Runtime: cross-platform, high performance ML inferencing
Port of Facebook's LLaMA model in C/C++
User-friendly AI Interface
C++ library for high performance inference on NVIDIA GPUs
A high-throughput and memory-efficient inference and serving engine
The official Python client for the Huggingface Hub
Protect and discover secrets using Gitleaks
High-performance neural network inference framework for mobile
Unified Model Serving Framework
Self-hosted, community-driven, local OpenAI compatible API
Lightweight inference library for ONNX files, written in C++
OpenVINO™ Toolkit repository
The deep learning toolkit for speech-to-text
Uncover insights, surface problems, monitor, and fine tune your LLM
MNN is a blazing fast, lightweight deep learning framework
Training and deploying machine learning models on Amazon SageMaker
Open-Source AI Camera. Empower any camera/CCTV
State-of-the-art diffusion models for image and audio generation
Easy-to-use deep learning framework with 3 key features
Efficient few-shot learning with Sentence Transformers
Bayesian inference with probabilistic programming
Pure C++ implementation of several models for real-time chatting
Implementation of model parallel autoregressive transformers on GPUs