Browse free open source Python LLM Inference Tools and projects below. Use the toggles on the left to filter open source Python LLM Inference Tools by OS, license, language, programming language, and project status.
FlashInfer: Kernel Library for LLM Serving
A high-throughput and memory-efficient inference and serving engine
Run Local LLMs on Any Device. Open-source
Ready-to-use OCR with 80+ supported languages
Everything you need to build state-of-the-art foundation models
State-of-the-art diffusion models for image and audio generation
Library for OCR-related tasks powered by Deep Learning
GPU environment management and cluster orchestration
Lightweight anchor-free object detection model
OpenMMLab Model Deployment Framework
Sparsity-aware deep learning inference runtime for CPUs
Standardized Serverless ML Inference Platform on Kubernetes
Bring the notion of Model-as-a-Service to life
The official Python client for the Huggingface Hub
Single-cell analysis in Python
A set of Docker images for training and serving models in TensorFlow
Uncover insights, surface problems, monitor, and fine tune your LLM
Database system for building simpler and faster AI-powered application
Visual Instruction Tuning: Large Language-and-Vision Assistant
Scripts for fine-tuning Meta Llama3 with composable FSDP & PEFT method
Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
Trainable models and NN optimization tools
AIMET is a library that provides advanced quantization and compression
Openai style api for open large language models
Images to inference with no labeling