Browse free open source LLM Inference tools and projects for Windows below. Use the toggles on the left to filter open source LLM Inference tools by OS, license, language, programming language, and project status.
Port of OpenAI's Whisper model in C/C++
Port of Facebook's LLaMA model in C/C++
Run Local LLMs on Any Device. Open-source
ONNX Runtime: cross-platform, high performance ML inferencing
C++ library for high performance inference on NVIDIA GPUs
Ready-to-use OCR with 80+ supported languages
User-friendly AI Interface
A high-throughput and memory-efficient inference and serving engine
High-performance neural network inference framework for mobile
The official Python client for the Huggingface Hub
Protect and discover secrets using Gitleaks
Self-hosted, community-driven, local OpenAI compatible API
OpenVINO™ Toolkit repository
The deep learning toolkit for speech-to-text
Lightweight inference library for ONNX files, written in C++
Easy-to-use deep learning framework with 3 key features
Uncover insights, surface problems, monitor, and fine tune your LLM
Library for OCR-related tasks powered by Deep Learning
Bring the notion of Model-as-a-Service to life
State-of-the-art diffusion models for image and audio generation
Implementation of model parallel autoregressive transformers on GPUs
Lightweight anchor-free object detection model
Single-cell analysis in Python
Unified Model Serving Framework
Pure C++ implementation of several models for real-time chatting