Browse free open source Python LLM Inference Tools and projects below. Use the toggles on the left to filter open source Python LLM Inference Tools by OS, license, language, programming language, and project status.
Run Local LLMs on Any Device. Open-source
Ready-to-use OCR with 80+ supported languages
A high-throughput and memory-efficient inference and serving engine
The official Python client for the Huggingface Hub
Unified Model Serving Framework
Bring the notion of Model-as-a-Service to life
Lightweight anchor-free object detection model
Uncover insights, surface problems, monitor, and fine tune your LLM
Training and deploying machine learning models on Amazon SageMaker
State-of-the-art diffusion models for image and audio generation
Efficient few-shot learning with Sentence Transformers
Implementation of model parallel autoregressive transformers on GPUs
A library for accelerating Transformer models on NVIDIA GPUs
Library for OCR-related tasks powered by Deep Learning
Visual Instruction Tuning: Large Language-and-Vision Assistant
Easiest and laziest way for building multi-agent LLMs applications
Easy-to-use Speech Toolkit including Self-Supervised Learning model
Single-cell analysis in Python
AIMET is a library that provides advanced quantization and compression
Python Package for ML-Based Heterogeneous Treatment Effects Estimation
Standardized Serverless ML Inference Platform on Kubernetes
A Pythonic framework to simplify AI service building
20+ high-performance LLMs with recipes to pretrain, finetune at scale
Trainable models and NN optimization tools
A lightweight vision library for performing large object detection