vLLM download | SourceForge.net

vLLM is a fast and easy-to-use library for LLM inference and serving. High-throughput serving with various decoding algorithms, including parallel sampling, beam search, and more.

Features

State-of-the-art serving throughput
Efficient management of attention key and value memory with PagedAttention
Continuous batching of incoming requests
Optimized CUDA kernels
Seamless integration with popular HuggingFace models
Tensor parallelism support for distributed inference

Project Samples

Project Activity

See All Activity >

License

Apache License V2.0

Follow vLLM

vLLM Web Site

Other Useful Business Software

Cut Data Warehouse Costs by 54%

Easily migrate from Snowflake, Redshift, or Databricks with free tools.

BigQuery delivers 54% lower TCO with exabyte scale and flexible pricing. Free migration tools handle the SQL translation automatically.

Start Free

Rate This Project

User Reviews

Be the first to post a review of vLLM!

Additional Project Details

Programming Language

Python

Related Categories

Python Large Language Models (LLM), Python LLM Inference Tool

Registered

2023-08-21

Similar Business Software

Gemini platform

Gemini platform is an enterprise AI development and machine learning platform from Google Cloud that enables organizations to build, customize, deploy, and manage AI agents and intelligent applications. Formerly known as Vertex AI, the platform provides access to more than 200 foundation models...

See Software for Gemini platform
LM-Kit.NET

LM-Kit.NET is a complete local AI runtime for .NET that lets engineering teams ship AI-powered features without cloud dependencies, per-token costs, or data leaving the network. Most .NET AI integrations stop at inference. LM-Kit.NET covers the full range of capabilities production...

See Software for LM-Kit.NET
Google AI Studio

Google AI Studio is a unified development platform that helps teams explore, build, and deploy applications using Google’s most advanced AI models. It brings text, image, audio, and video models together in one interactive playground. With vibe coding, developers can use natural language to...

See Software for Google AI Studio
Compute with Hivenet

Hivenet Compute gives you on-demand RTX 5090 and RTX Pro 6000 GPUs, plus vCPU instances, billed per second while they run. Hivenet owns and operates the infrastructure, with data centres in France, the UAE and the US, so EU teams can keep workloads and data in Europe. Use it for training and...

See Software for Compute with Hivenet
vLLM

vLLM is a high-performance library designed to facilitate efficient inference and serving of Large Language Models (LLMs). Originally developed in the Sky Computing Lab at UC Berkeley, vLLM has evolved into a community-driven project with contributions from both academia and industry. It offers...

See Software for vLLM
Runpod

Runpod offers a cloud-based platform designed for running AI workloads, focusing on providing scalable, on-demand GPU resources to accelerate machine learning (ML) model training and inference. With its diverse selection of powerful GPUs like the NVIDIA A100, RTX 3090, and H100, Runpod supports...

See Software for Runpod