Browse free open source LLM Inference tools and projects for Linux below. Use the toggles on the left to filter open source LLM Inference tools by OS, license, language, programming language, and project status.
Port of OpenAI's Whisper model in C/C++
Port of Facebook's LLaMA model in C/C++
Run Local LLMs on Any Device. Open-source
ONNX Runtime: cross-platform, high performance ML inferencing
User-friendly AI Interface
C++ library for high performance inference on NVIDIA GPUs
Self-hosted, community-driven, local OpenAI compatible API
A high-throughput and memory-efficient inference and serving engine
Protect and discover secrets using Gitleaks
OpenVINO™ Toolkit repository
State-of-the-art diffusion models for image and audio generation
Open standard for machine learning interoperability
Build Production-ready Agentic Workflow with Natural Language
C++ implementation of ChatGLM-6B & ChatGLM2-6B & ChatGLM3 & GLM4(V)
High-performance neural network inference framework for mobile
The deep learning toolkit for speech-to-text
LLM training code for MosaicML foundation models
Easy-to-use deep learning framework with 3 key features
Open-Source AI Camera. Empower any camera/CCTV
Deep Learning API and Server in C++14 support for Caffe, PyTorch
A lightweight vision library for performing large object detection
Optimizing inference proxy for LLMs
AIMET is a library that provides advanced quantization and compression
A set of Docker images for training and serving models in TensorFlow
Python Package for ML-Based Heterogeneous Treatment Effects Estimation