LightLLM - Browse /v1.2.0 at SourceForge.net

The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
README.md	2026-08-04	29.1 kB	0
v1.2.0 source code.tar.gz	2026-08-04	3.8 MB	0
v1.2.0 source code.zip	2026-08-04	4.7 MB	3
Totals: 3 Items		8.6 MB	3

This release introduces RL serving with verl, disaggregated ViT serving, a hybrid-cache architecture for linear-attention models, and a multi-level cache system with cache-aware PD. It also delivers broad model-performance improvements—including expanded MTP speculative decoding and optimizations across prefill, attention, MoE, quantization, and distributed inference—together with extensive stability and correctness fixes across scheduling, caching, multimodal processing, APIs, and long-running services.

Highlights

Reinforcement Learning Serving with verl

LightLLM can now serve as the rollout and inference backend for verl-based reinforcement learning workflows. A dedicated RL control plane supports online model-weight updates through distributed communication, tensors, or CUDA IPC, together with cache flushing, request aborts, generation pause/resume, and GPU-memory release/resume. The integration also includes routing-data capture for MoE training. #1298

Disaggregated ViT Serving

Vision encoding can now run as an independent visual_only service on dedicated GPU nodes. LLM servers use a proxy mode to discover registered visual workers through the config server, dispatch image encoding remotely, and retrieve the resulting embeddings through a shared AFS/Redis-backed cache. This separates ViT compute from language-model inference so the two tiers can be deployed and scaled independently. The release also adds configurable ViT attention backends and independent controls for visual and audio modules. #1234 #1191 #1229

Hybrid Radix Cache with Large and Small Pages

A new dual-granularity Hybrid Radix Cache is introduced for hybrid full-attention and linear-attention models such as Qwen3.5 and Qwen3-Next:

Large pages represent multi-block, reusable prefix checkpoints and keep the full-attention KV data together with the corresponding linear-attention convolution and SSM states.
Small pages preserve fine-grained or fragmented prefix tails, avoiding coarse page boundaries from limiting cache reuse.
Small-page entries can be consolidated into large-page checkpoints and integrated with CPU cache/offload, allowing the cache to retain fine-grained matching while managing recurrent states efficiently.

The design addresses the different storage granularities required by token-level KV cache and recurrent linear-attention state, enabling effective prefix caching for models that combine full and linear attention. #1265 #1422

Multi-Level Cache and Cache-Aware PD

The cache hierarchy now spans GPU, CPU, and disk, with support for quantized CPU KV cache, FP8/INT8 KV formats, NUMA-aware CPU page placement, and faster cache initialization. PD disaggregation gains a cache-aware scheduling path, upgraded NIXL KV transfer, Qwen3.5 support, and stronger lifecycle, health-check, disconnect, and multimodal handling. #997 #1098 #1379 #1399 #1412