Browse free open source LLM Inference tools and projects for Windows below. Use the toggles on the left to filter open source LLM Inference tools by OS, license, language, programming language, and project status.
Port of OpenAI's Whisper model in C/C++
Port of Facebook's LLaMA model in C/C++
Run Local LLMs on Any Device. Open-source
User-friendly AI Interface
ONNX Runtime: cross-platform, high performance ML inferencing
Self-hosted, community-driven, local OpenAI compatible API
A high-throughput and memory-efficient inference and serving engine
Ready-to-use OCR with 80+ supported languages
OpenVINO™ Toolkit repository
State-of-the-art diffusion models for image and audio generation
Protect and discover secrets using Gitleaks
C++ library for high performance inference on NVIDIA GPUs
High-performance neural network inference framework for mobile
C++ implementation of ChatGLM-6B & ChatGLM2-6B & ChatGLM3 & GLM4(V)
Library for OCR-related tasks powered by Deep Learning
Open-Source AI Camera. Empower any camera/CCTV
Training and deploying machine learning models on Amazon SageMaker
Implementation of model parallel autoregressive transformers on GPUs
A RWKV management and startup tool, full automation, only 8MB
A set of Docker images for training and serving models in TensorFlow
The deep learning toolkit for speech-to-text
Optimizing inference proxy for LLMs
OpenMMLab Model Deployment Framework
Open standard for machine learning interoperability
The Triton Inference Server provides an optimized cloud