DownloadvLLM is a fast and easy-to-use library for LLM inference and serving. High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more.
Features
- State-of-the-art serving throughput
- Efficient management of attention key and value memory with PagedAttention
- Continuous batching of incoming requests
- Optimized CUDA kernels
- Seamless integration with popular HuggingFace models
- Tensor parallelism support for distributed inference
Project Samples
Categories
Large Language Models (LLM)License
Apache License V2.0Follow VLLM
You Might Also Like
Stop messing with tools that aren’t designed to amplify volunteer programs. With VolunteerMatters, it’s a delight to manage everything in one place.
Rate This Project
Login To Rate This Project
User Reviews
Be the first to post a review of VLLM!