AReal Files

Release Note

We're excited to announce AReaL v0.3.3, which stabilizes training for larger dense models with extended context lengths. This release includes essential improvements and new algorithms to deliver the best out-of-the-box experience for users.

Enhanced Parallelism Support - Added hybrid parallelism with FSDP backend: tensor parallelism, Ulysses sequence parallelism, and sequence-parallel activation checkpointing - Zero conversion required – use ANY Hugging Face model directly - Memory efficient – support for long context lengths with reduced GPU activation memory - Usage: allocation_mode=sglang:d8 + fsdp:d2c2t2

New Algorithm Features - PPO with clip higher - Dynamic sampling with variable batch sizes - Over-length penalty mechanism - Decoupled mean/std computation for advantage estimation

Hardware Compatibility - We are ready to support additional hardware backends beyond NVIDIA GPUs (more announcements coming soon!)

What's Changed

• FEAT: Decoupled CLIP ratio (DAPO Trick-I) by @ZiyiTsang in https://github.com/inclusionAI/AReaL/pull/285
• Add agent-related logging logic in ppo actor & Update notebook example by @samjia2000 in https://github.com/inclusionAI/AReaL/pull/290
• FEAT: Dynamic_Sampling(DAPO Trick-II) by @ZiyiTsang in https://github.com/inclusionAI/AReaL/pull/294
• refactor: refactor examples structure, make fsdp and ulysses use independent device meshes by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/297
• doc: update the doc of using ulysses sp by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/298
• [TEST] megatron dcp save load test by @fishcrap in https://github.com/inclusionAI/AReaL/pull/306
• fix doc: update package installation method within container by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/307
• refactor: group examples according to application by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/305
• fix: add the missing group argument in data redistribution by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/311
• use sourceTensor.detach().clone() rather than torch.tensor(sourceTensor) by @CormickKneey in https://github.com/inclusionAI/AReaL/pull/308
• add countdown example by @samjia2000 in https://github.com/inclusionAI/AReaL/pull/299
• Support tensor parallelism for FSDP engine by @rchardx in https://github.com/inclusionAI/AReaL/pull/309
• FEAT: Overlong_Reward_Penalty (DAPO Trick-III) by @ZiyiTsang in https://github.com/inclusionAI/AReaL/pull/295
• chore: add engine IDs to differentiate different ranks in logs by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/314
• In remote engine, find sglang server using experiment name and trial name by @samjia2000 in https://github.com/inclusionAI/AReaL/pull/301
• [Bug Fix] Fix server_idx initialization in RemoteSGLangEngine by @nuzant in https://github.com/inclusionAI/AReaL/pull/318
• chore: Fix signatures of rollout.initialize in examples by @nuzant in https://github.com/inclusionAI/AReaL/pull/319
• chore: amend the should_accept argument in rollout_batch with docs by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/316
• fix: The shape of attention_mask itself gets changed when removing pads by @jwhj in https://github.com/inclusionAI/AReaL/pull/325
• Fix the gradient norm clipping for FSDP engine by @rchardx in https://github.com/inclusionAI/AReaL/pull/320
• chore: raise error when using slurm with apptainer and images are not specified. by @nuzant in https://github.com/inclusionAI/AReaL/pull/329
• Apply sequence parallel to LayerNorm/RMSNorm layers by @rchardx in https://github.com/inclusionAI/AReaL/pull/330
• chore: add ci to close stale issues by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/332
• Import missing AllocationMode by @rchardx in https://github.com/inclusionAI/AReaL/pull/333
• [Feat] Add device agnostic feature by @lowdy1 in https://github.com/inclusionAI/AReaL/pull/327
• Update pre-commit hooks and rerun against all the files by @rchardx in https://github.com/inclusionAI/AReaL/pull/334
• Decouple the mean&std advantage normalization (Trick Dr. GRPO and LitePPO) by @ZiyiTsang in https://github.com/inclusionAI/AReaL/pull/303
• fix fsdp engine: qwen3 TP q/k norm wrapping, gradient clipping, the scale of grad norm, and sft scripts by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/335
• chore: preventing CI to close stale PRs by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/337
• fix: revert the order of evaluation and recover in entrypoints, fix all unit-tests by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/323
• [device agnostic] fix examples with the usage of current_platform by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/338
• refactor: move should_accept to the submit method instead of the wait method by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/339
• [Bug Fix] Fix loading Qwen2 1.5B with MegatronEngine and mbridge by @nuzant in https://github.com/inclusionAI/AReaL/pull/341
• [Feature] Add deterministic option for MegatronEngine by @nuzant in https://github.com/inclusionAI/AReaL/pull/340
• [device agnostic] chore: replace cuda with current_platform.device_type by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/336
• Fix a bug of embeds_token for VL models; Refine some YAML configuration files by @rchardx in https://github.com/inclusionAI/AReaL/pull/342
• doc: update readme by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/346
• doc: update readme by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/347
• Fix examples by @fishcrap in https://github.com/inclusionAI/AReaL/pull/344
• [Tests] Fix some bugs in examples, add a unittest that runs every examples for one step. by @nuzant in https://github.com/inclusionAI/AReaL/pull/345
• Bump v0.3.3 by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/349

New Contributors

• @CormickKneey made their first contribution in https://github.com/inclusionAI/AReaL/pull/308
• @rchardx made their first contribution in https://github.com/inclusionAI/AReaL/pull/309
• @jwhj made their first contribution in https://github.com/inclusionAI/AReaL/pull/325
• @lowdy1 made their first contribution in https://github.com/inclusionAI/AReaL/pull/327

Full Changelog: https://github.com/inclusionAI/AReaL/compare/v0.3.2...v0.3.3