Release Highlights

• Prefill disaggregation is now supported in initial support in Ray Serve LLM (#53092). This is critical for production LLM serving use cases.
• Ray Data features a variety of performance improvements (locality-based scheduling, non-blocking execution) as well as improvements to observability, preprocessors, and other stability fixes.
• Ray Serve now features custom request routing algorithms, which is critical for high throughput traffic for large model use cases.

Ray Libraries

Ray Data

🎉 New Features: - Add save modes support to file data sinks (#52900) - Added flattening capability to the Concatenator preprocessor to support output vectorization use cases (#53378)

💫 Enhancements: - Re-enable Actor locality-based scheduling. This PR also improves algorithms for ranking the locations for the bundle. (#52861) - Disable blocking pipeline by default until Actor Pool fully scales up to min actors (#52754) - Progress bar and dashboard improvements to show name of partial functions properly(#52280)

🔨 Fixes: - Make Ray Data from_torch respect Dataset len (#52804) - Fixing flaky aggregation test (#53383) - Fix race condition bug in fault tolerance by disabling on_exit hook (#53249) - Fix move_tensors_to_device utility for the list/tuple[tensor] case (#53109) - Fix ActorPool scaling to avoid scaling down when the input queue is empty (#53009) - Fix internal queues accounting for all Operators w/ an internal queue (#52806) - Fix backpressure for FileBasedDatasource. This fixes potential OOMs for workloads using FileBasedDatasources (#52852)

📖 Documentation: - Fix working code snippets (#52748) - Improve AggregateFnV2 docstrings and examples (#52911) - Improved documentation for vectorizers and API visibility in Data (#52456)

Ray Train

🎉 New Features: - Added support for configuring Ray Train worker actor runtime environments. (#52421) - Included Grafana panel data in Ray Train export for improved monitoring. (#53072) - Introduced a structured logging environment variable to standardize log formats. (#52952) - Added metrics for TrainControllerState to enhance observability. (#52805)

💫 Enhancements: - Logging of controller state transitions to aid in debugging and analysis. (#53344) - Improved handling of Noop scaling decisions for smoother scaling logic. (#53180)

🔨 Fixes: - Improved move_tensors_to_device utility to correctly handle list / tuple of tensors. (#53109) - Fixed GPU transfer support for non-contiguous tensors. (#52548) - Increased timeout in test_torch_device_manager to reduce flakiness. (#52917)

📖 Documentation: - Added a note about PyTorch DataLoader’s multiprocessing and forkserver usage. (#52924) - Fixed various docstring format and indentation issues. (#52855, [#52878]) - Removed unused "configuration-overview" documentation page. (#52912) - General typo corrections. (#53048)

🏗 Architecture refactoring: - Deduplicated ML doctest runners in CI for efficiency. (#53157) - Converted isort configuration to Ruff for consistency. (#52869) - Removed unused PARALLEL_CI blocks and combined imports. (#53087, [#52742])

Ray Tune

💫 Enhancements: - Updated test_train_v2_integration to use the correct RunConfig. (#52882)

📖 Documentation: - Replaced session.report with tune.report and corrected import paths. (#52801) - Removed outdated graphics cards reference in docs. (#52922) - Fixed various docstring format issues. (#52879)

Ray Serve

🎉 New Features: - Added support for implementing custom request routing algorithms. (#53251) - Introduced an environment variable to prioritize custom resources during deployment scheduling. (#51978)

💫 Enhancements: - The ingress API now accepts a builder function in addition to an ASGI app object. (#52892)

🔨 Fixes: - Fixed runtime_env validation for py_modules. (#53186) - Disallowed special characters in Serve deployment and application names. (#52702) - Added a descriptive error message when a deployment name is not found. (#45181)

📖 Documentation: - Updated the guide on serving models with Triton Server in Ray Serve. - Added documentation for custom request routing algorithms.

Ray Serve/Data LLM

🎉 New Features: - Added initial support for prefill decode disaggregation (#53092) - Expose vLLM Metrics to serve.llm API (#52719) - Embedding API (#52229)

💫 Enhancements: - Allow setting name_prefix in build_llm_deployment (#53316) - Minor bug fix for 53144: stop tokens cannot be null (#53288) - Add missing repetition_penalty vLLM sampling parameter (#53222) - Mitigate the serve.llm streaming overhead by properly batching stream chunks (#52766) - Fix test_batch_vllm leaking resources by using larger wait_for_min_actors_s

🔨 Fixes: - LLMRouter.check_health() should check LLMServer.check_health() (#53358) - Fix runtime passthrough and auto-executor class selection (#53253) - Update check_health return type (#53114) - Bug fix for duplication of <bos> token (#52853) - In stream batching, first part of the stream was always consumed and not streamed back from the router (#52848)

RLlib

🎉 New Features: - Add GPU inference to offline evaluation. (#52718)

💫 Enhancements: - Do-over of examples for connector pipelines. (#52604) - Cleanup of meta learning classes and examples. (#52680)

🔨 Fixes: - Fixed weight synching in offline evaluation. (#52757) - Fixed bug in split_and_zero_pad utility function (related to complex structures vs simple values or np.arrays). (#52818)

Ray Core

💫 Enhancements: - uv run integration is now enabled by default, so you don't need to set the RAY_RUNTIME_ENV_HOOK any more (#53060) - Record gcs process metrics (#53171)

🔨 Fixes: - Improvements for using RuntimeEnv in the Job Submission API. (#52704) - Close unused pipe file descriptor of child processes of Raylet (#52700) - Fix race condition when canceling task that hasn't started yet (#52703) - Implement a thread pool and call the CPython API on all threads within the same concurrency group (#52575) - cgraph: Fix execution schedules with collective operations (#53007) - cgraph: Fix scalar tensor serialization edge case with serialize_to_numpy_or_scalar (#53160) - Fix the issue where a valid RestartActor rpc is ignored (#53330) - Fix reference counter crashes during worker graceful shutdown (#53002)

Dashboard

🎉 New Features: - train: Add dynolog for on-demand GPU profiling for Torch training (#53191)

💫 Enhancements: - Add configurability of 'orgId' param for requesting Grafana dashboards (#53236)

🔨 Fixes: - Fix Grafana dashboards dropdowns for data and train dashboard (#52752) - Fix dashboard for daylight savings (#52755)

Ray Container Images

💫 Enhancements: - Upgrade h11 (#53361), requests, starlette, jinja2 (#52951), pyopenssl and cryptography (#52941) - Generate multi-arch image indexes (#52816)

Docs

🎉 New Features: - End-to-end example: Entity recognition with LLMs (#52342) - new end-to-end example - End-to-end example: xgboost tutorial (#52383) - End-to-end tutorial for audio transcription and LLM as judge curation (#53189)

💫 Enhancements: - Adds pydoclint to pre-commit (#52974)

Thanks!

Thank you to everyone who contributed to this release!

@NeilGirdhar, @ok-scale, @JiangJiaWei1103, @brandonscript, @eicherseiji, @ktyxx, @MichalPitr, @GeneDer, @rueian, @khluu, @bveeramani, @ArturNiederfahrenhorst, @c8ef, @lk-chen, @alanwguo, @simonsays1980, @codope, @ArthurBook, @kouroshHakha, @Yicheng-Lu-llll, @jujipotle, @aslonnie, @justinvyu, @machichima, @pcmoritz, @saihaj, @wingkitlee0, @omatthew98, @can-anyscale, @nadongjun, @chris-ray-zhang, @dizer-ti, @matthewdeng, @ryanaoleary, @janimo, @crypdick, @srinathk10, @cszhu, @TimothySeah, @iamjustinhsu, @mimiliaogo, @angelinalg, @gvspraveen, @kevin85421, @jjyao, @elliot-barn, @xingyu-long, @LeoLiao123, @thomasdesr, @ishaan-mehta, @noemotiovon, @hipudding, @davidxia, @omahs, @MengjinYan, @dengwxn, @MortalHappiness, @alhparsa, @emmanuel-ferdman, @alexeykudinkin, @KunWuLuan, @dev-goyal, @sven1977, @akyang-anyscale, @GokuMohandas, @raulchen, @abrarsheikh, @edoakes, @JoshKarpel, @bhmiller, @seanlaii, @ruisearch42, @dayshah, @Bye-legumes, @petern48, @richardliaw, @rclough, @israbbani, @jiwq