Browse free open source Python LLM Inference Tools and projects below. Use the toggles on the left to filter open source Python LLM Inference Tools by OS, license, language, programming language, and project status.
Run Local LLMs on Any Device. Open-source
A high-throughput and memory-efficient inference and serving engine
Ready-to-use OCR with 80+ supported languages
Library for OCR-related tasks powered by Deep Learning
State-of-the-art diffusion models for image and audio generation
LLM training code for MosaicML foundation models
Bring the notion of Model-as-a-Service to life
A lightweight vision library for performing large object detection
Optimizing inference proxy for LLMs
AIMET is a library that provides advanced quantization and compression
A set of Docker images for training and serving models in TensorFlow
Python Package for ML-Based Heterogeneous Treatment Effects Estimation
Implementation of model parallel autoregressive transformers on GPUs
The unofficial python package that returns response of Google Bard
Unified Model Serving Framework
Deep learning optimization library: makes distributed training easy
LMDeploy is a toolkit for compressing, deploying, and serving LLMs
Easiest and laziest way for building multi-agent LLMs applications
OpenMMLab Model Deployment Framework
Framework for Accelerating LLM Generation with Multiple Decoding Heads
Lightweight anchor-free object detection model
Lightweight Python library for adding real-time multi-object tracking
State-of-the-art Parameter-Efficient Fine-Tuning
A unified framework for scalable computing
Replace OpenAI GPT with another LLM in your app