GPUStack v2.3.0

🚀 Model Catalog Updates

LLM: DeepSeek-V4.1-Flash-1017, MiMo-V2.6-Pro-MOPD, GLM-5.3-Flash, Qwen3.8-27B, GLM-5.3, Muse-Glimmer-30B, Kimi-K3.

✨ New Features & Enhancements

Model Catalog & Inference Backends

• OTA Content Updates: Automatically update the model catalog, Community Backend Marketplace, and built-in backend version list without upgrading or restarting GPUStack. Supports configurable update intervals, on-demand refresh, custom URL sources, model catalog YAML import, and embedded content for offline use (#5518.
• Improve model catalog search with tokenized, separator-insensitive matching (#5810).

Model Deployment & Lifecycle

• Prefill/Decode Disaggregation: Configure prefill and decode replicas independently, with managed routing, scheduling, and deployment status. Supports vLLM, SGLang, and Ascend deployment modes, with compatible extended KV cache configurations (#5184).
• GPU Slicing: Deploy models and GPU instances on sliced GPU resources with configurable resource limits and usage metering for GPU instance allocations (#5192, #6054).
• Scheduled Scaling: Automatically adjust model replicas according to configured time windows (#5066).
• Deployment Version History & Rollback: Track deployment configuration revisions, compare changes, and roll back to a previous version (#6152).
• YAML Import & Export: Export and import model deployment configurations for backup, migration, and reuse (#5755).
• Support custom container images with the built-in backends, retaining their scheduling and distributed inference capabilities (#6208).
• Allow changing the model source when cloning a deployment (#5841).
• Use /health for vLLM/SGLang readiness checks to avoid marking instances ready before initialization completes (#6203).
• Show the container exit code when a model instance enters an error state (#4217).
• Set CUDA_DEVICE_ORDER=PCI_BUS_ID by default in NVIDIA environments for consistent GPU ordering (#6041).

Shared KV Cache

• Shared KV Cache Services: Manage cache independently of inference processes, allowing replicas to reuse cached prefixes and extend cache capacity with L2 storage (#5805).
• XSKY MeshFusion Integration: Reuse KV cache across model deployments and replicas, with XSKY MeshFusion Store providing shared L2 storage for cache reuse across nodes (#6088).
• Support LMCache MP P2P for KV cache sharing across workers (#6186).
• Support in-process extended KV cache for vLLM on Ascend (#5591).

Routing, Gateway & API

• Gateway Load Balancing: Support weighted routing, round-robin, least-inflight-request, and session-affinity policies, with health handling and fallback targets (#6265).
• Decision Service Routing: Integrate with Jev-compatible decision services to select models based on requests and model capabilities (#6312).
• Support per-endpoint protocol selection for OpenAI Chat Completions, Responses, and Anthropic Messages, including native /v1/messages passthrough and the DeepSeek Responses API (#6010, #5286, #6026).
• Support custom base URLs for Anthropic-compatible providers (#5314).
• Support mapping route aliases to upstream model names containing / (#6180).
• Support video generation API routing, including /v1/videos/sync (#6328).
• Gateway-side API Key Authentication: Optimize credential verification and cache authorization decisions to reduce per-request server calls and improve resilience to brief authentication service interruptions (#5991, #6003).
• Filter /v1/models by API key permissions, returning only models the key can access (#6323).
• Add optional tool-call pairing validation at the gateway. Enable toolCallValidation: strict to reject incomplete or mismatched tool / tool_calls messages (#6210).
• Add segmented proxy and tunnel timeouts, distinguishing the wait for the first response chunk from stalls during streaming and reporting explicit timeout errors (#6012).
• Support configuring minimum and maximum TLS versions for Higress through environment variables (#6182).

GPU Instance Service

• Support viewing GPU instance resource utilization (#5616).
• Enable token authentication for the Jupyter Notebook template, with authenticated access links in the Web UI (#5757).
• Support NVIDIA MIG instance type mapping to Kubernetes nvidia.com/mig-* resources (#5777).

Benchmarking

• Full Load Curves in One Benchmark: Sweep multiple concurrency or request-rate points, with Max Throughput, latency SLO, and Custom modes, per-point results, and recommended operating points (#5765).
• Support the ShareGPT dataset and improve random seed configuration for benchmark workloads (#5980, #5894).
• Support cloning benchmarks to reuse existing configurations (#5778).

Clusters, Monitoring & Logs

• Add SHUIHUA FUTURE cloud provider support for provisioning and managing GPU worker pools with tunnel-based inference access (29f9f73a).
• Support configuring and optionally deploying Operator-managed components, including NFD, Kueue, and CSI drivers, through the Helm chart (#6011).
• Allow skipping CPU workers during Kubernetes cluster registration (#5974).
• Improve worker registration with self-signed TLS certificates, and support insecure_tls in worker configuration to explicitly skip server certificate verification (#5006, #5992).
• Add a model category label to the gpustack:model_info metric (#5940).
• Add Higress Gateway metrics collection for the built-in Higress dashboards (#6146).
• Improve model log pagination and identify log directories by model instance name (#6126, #5858).

UI/UX

• Add reasoning_effort configuration and display cached tokens and cache hit rate in Playground (#6069, #6241).
• Allow stopping streaming TTS generation in the audio Playground (#5570).
• Serve precompressed UI assets with cache headers to reduce transfer size and improve asset refresh after upgrades (#6034).
• Bundle multilingual documentation accessible from GPUStack for offline use (#6484).
• Allow . in passwords and display the configured SSO provider name on the login page (#5952, #6163).

🐛 Bug Fixes

Model Deployment & Inference

• Fixed vLLM 0.25 / 0.26 MP multi-node startup failures caused by ZMQ port conflicts and improved preservation of initial failure logs across subordinate restarts (#6019).
• Fixed MP distributed deployment parameter compatibility with vLLM 0.28 and later (#6221).
• Fixed SGLang --tp not taking effect (#5966).
• Fixed SGLang /v1/responses returning 404 due to a missing --served-model-name argument (#6386).
• Fixed empty backend_version not being treated as Auto, causing vLLM image resolution to fail (#6014).
• Fixed Ascend 950PR device detection (#6148).
• Fixed download progress reaching 99% before most model data was downloaded, and completed nodes showing 0% progress during distributed deployments (#6150, #5666).
• Fixed non-streaming inference continuing on the backend after the client disconnects (#6286).

Gateway & Model Providers

• Fixed uneven load distribution across model instances (#5586).
• Fixed adding Qwen3.7-Max and Kimi-K3 routes through the Qwen provider (#5402, #5969).
• Fixed route target deletion failures and add-targets returning 500 when updating an existing target (#5979, #6227).
• Fixed Ingress resources with empty destinations when no backend is running, and requests incorrectly returning 404 instead of 503 when a worker becomes unavailable (#6098, #6030).
• Fixed model routes not recovering after a TUNNEL worker reconnects (#6188).
• Fixed missing model_route_id in gateway metrics and valid model-mapper rules being deleted on server startup when GPUStack and Higress use different namespaces (#5999, #6454).
• Fixed large, multi-round Claude Code requests timing out with 503 during gateway request processing (#6217).
• Fixed the gateway hanging on upstream errors with an empty response body instead of forwarding the status code (#6304).

GPU Instances, Clusters & Database

• Fixed missing /dev/davinci* devices in Ascend GPU instance containers (#5732).
• Fixed GPU instance containers exposing extra GPUs when only one GPU was allocated (#5853).
• Fixed SSH TCP forwarding, restoring VS Code connectivity to GPU instances (#5801).
• Fixed Kueue queue lookup failures for GPU model deployments on Kubernetes (#6002).
• Fixed Kubernetes resource names containing uppercase characters being replaced with opaque names (#5597).
• Fixed editing upgraded Kubernetes clusters being blocked by data-directory volume mount validation (#6145).
• Fixed cluster registration resetting the selected GPU vendor to NVIDIA instead of preserving the cluster's registered vendor (#5977).
• Fixed cloud worker pools failing to provision nodes when batch_size is omitted in API requests (#6128).
• Fixed database connection recovery after PostgreSQL failover and connection URL query parameters being silently discarded (#6105, #6106).

Logs, Monitoring & Usage

• Fixed distributed deployment log downloads containing only the main node's logs (#5662).
• Fixed large log downloads failing and blank logs after adopting existing running containers during an upgrade (#6125, #5988).
• Fixed missing UI errors when inference image pulls fail on Kubernetes (#5869).
• Fixed metrics aggregation with the vLLM Rust frontend's OpenMetrics output and stale metrics remaining after model instances are removed (#6215, #6282).
• Fixed allocated VRAM indicators omitting distributed deployment allocations (#5654).
• Fixed token usage exports containing only the current page, allowing all matching data to be exported (#6013).
• Fixed Anthropic input token accounting excluding cached tokens, making input and total usage consistent with the OpenAI protocol (#6405).
• Fixed incorrect grouped token usage tooltips and model filter entries becoming inconsistent after repeated scrolling (#6326, #6457).

Benchmarking & UI/UX

• Fixed missing benchmark results for some models and benchmark failures with self-signed server certificates (#5910, #6102).
• Fixed benchmark model filtering returning no results and incorrect request delay units (#5915, #6201).
• Fixed Playground Max Tokens parameter handling (#5909).
• Fixed initial admin password changes failing after refreshing the forced password change page (#5954).
• Fixed overlapping dashboard usage charts on narrow screens and misaligned GPU/VRAM rows in the workers table (#5976, #5978).

🔧 Built-in Inference Backend Updates

The following versions are newly bundled with v2.3.0. Additional backend versions can be delivered through OTA updates.

New Arrival

• CUDA 13.0
• vLLM 0.32.0 / 0.31.0 / 0.30.0 / 0.29.0 / 0.27.1
• SGLang 0.5.22 / 0.5.21 / 0.5.18
• CUDA 12.9
• vLLM 0.32.0 / 0.31.0 / 0.29.0 / 0.27.1
• SGLang 0.5.18
• ROCm 7.2
• vLLM 0.32.0 / 0.31.0 / 0.30.0 / 0.29.0 / 0.27.1
• SGLang 0.5.22 / 0.5.21 / 0.5.18
• CANN 9.2
• vLLM 0.30.0 (310P / 910B / 950 / A3)
• CANN 9.1
• vLLM 0.30.0 / 0.27.1 / 0.23.0 (310P / 910B / 950 / A3)
• CANN 9.0
• SGLang 0.5.20 / 0.5.18 (910B / A3)