User-friendly AI Interface
Openai style api for open large language models
Port of OpenAI's Whisper model in C/C++
Run Local LLMs on Any Device. Open-source
Port of Facebook's LLaMA model in C/C++
A high-throughput and memory-efficient inference and serving engine
OpenVINO™ Toolkit repository
ONNX Runtime: cross-platform, high performance ML inferencing
Self-hosted, community-driven, local OpenAI compatible API
High-performance neural network inference framework for mobile
Protect and discover secrets using Gitleaks
Everything you need to build state-of-the-art foundation models
Ready-to-use OCR with 80+ supported languages
MNN is a blazing fast, lightweight deep learning framework
C++ library for high performance inference on NVIDIA GPUs
FlashInfer: Kernel Library for LLM Serving
Bayesian inference with probabilistic programming
Pure C++ implementation of several models for real-time chatting
Trainable, memory-efficient, and GPU-friendly PyTorch reproduction
An MLOps framework to package, deploy, monitor and manage models
Library for serving Transformers models on Amazon SageMaker
An Open-Source Programming Framework for Agentic AI
LMDeploy is a toolkit for compressing, deploying, and serving LLMs
Bring the notion of Model-as-a-Service to life
Open standard for machine learning interoperability