AReal Files

Release Note

AReaL has been refactored from the legacy realhf codebase to the new areal codebase. These two directories are now independent, and our future development will focus primarily on the lightweight areal directory.

Major changes in v0.3.1 for the areal directory: + Added support for RL with Megatron 5D parallelism based on Megatron Core 0.13.1. We can now fine-tune large MoE models with AReaL. We also optimized weight loading and saving of Megatron models to the Hugging Face format, achieving 20x and 5x speedup for loading and saving respectively. + Added support for writing agentic RL workflows with the OpenAI client. Writing agentic RL is as easy as writing a standard agent! + Added support for Ulysses sequence parallelism with FSDP to reduce peak memory usage. + Added support for dp-attention, cross-node TP, and expert parallelism with SGLang. + Added support for automatic failover. + Created Jupyter notebook tutorials.

What's Changed

• Warning in the doc: How to run a synchronous configuration asynchronously. by @xssstory in https://github.com/inclusionAI/AReaL/pull/222
• chore: update stream notebook, issue template, and contribution guide by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/227
• [Feature] [refactor] Slightly refactor inference engine IO data structures by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/230
• [doc] Update gsm8k example hyperparameters by @EnderXie23 in https://github.com/inclusionAI/AReaL/pull/228
• [fix] Fix the unit-tests hanging bug by terminating the rollout thread by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/231
• ci: migrate to isolated runners by @futrime in https://github.com/inclusionAI/AReaL/pull/229
• [feature] support fault recovery and rollout-only evaluation by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/234
• [fix] Fix a minor iteration logic when using group_adv_norm by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/225
• [fix] Fix remote name error in gsm8k_grpo.yaml by @EnderXie23 in https://github.com/inclusionAI/AReaL/pull/235
• test(areal): SFT integration tests by @futrime in https://github.com/inclusionAI/AReaL/pull/233
• Update requirements.txt by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/237
• test(grpo): add GRPO integration tests by @futrime in https://github.com/inclusionAI/AReaL/pull/239
• [fix] Fix the name mismatch error of NCCL weight update for VLM and timeout error for computing rewards. by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/236
• [Fix] Fix the launcher errors of Ray/SLURM by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/242
• chore: add autoflake CI by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/245
• [fix] Add verbose messages for CI pytest by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/247
• [feat] Fix rollout completion order and allow stats logging during workflow execution. by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/246
• feat: support openai-compatible rollout and add an unittest for prepare_mb_list by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/248
• refactor: remove areal's dependency on realhf by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/249
• [FEAT] Support Variable Shape of Multi-Modal Inputs for VLM Training by @JamesKrW in https://github.com/inclusionAI/AReaL/pull/244
• [fix] remove the dependency of realhf in areal by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/252
• doc: add for writing workflows with the openai-compatible client by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/254
• chore: highlight wechat in readme by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/255
• [experimental] [feat] add megatron checkpointer and accelerate megatron weight loading by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/253
• fix: fix incorrect imports from realhf that causes statistics naming error by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/262
• [experimental] feat: megatron 5d parallel forward, reliable reward process executor, max length of dataset by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/263
• add search agent jupyter notebook example by @samjia2000 in https://github.com/inclusionAI/AReaL/pull/264
• fix: fix ci unit-test after gh runner recovers by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/268
• fix: replace LLMRequest with ModelRequest in the notebook by @ZiyiTsang in https://github.com/inclusionAI/AReaL/pull/271
• fix: replace LLMRequest with ModelRequest and format the asearcher notebook by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/272
• fix: fix the parsing logic of LLM_SERVER_ONLY allocation mode by @GurrenLagann97 in https://github.com/inclusionAI/AReaL/pull/265
• feat: support sglang cross-node TP and dp-attention with slurm by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/274
• [Doc] Fix a typo in a figure by @nuzant in https://github.com/inclusionAI/AReaL/pull/276
• feat: add megatron SFT example by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/275
• feat: support sglang cross-node ep and dp_attn with all launchers by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/277
• feat: support ulysses sequence parallel for FSDP by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/278
• fix: revert setting context parallel size when using ulysses by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/279
• feat: add a megatron grpo example by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/281
• Bump to v0.3.1 by @garrett4wade in https://github.com/inclusionAI/AReaL/pull/283

New Contributors

• @EnderXie23 made their first contribution in https://github.com/inclusionAI/AReaL/pull/228
• @JamesKrW made their first contribution in https://github.com/inclusionAI/AReaL/pull/244
• @ZiyiTsang made their first contribution in https://github.com/inclusionAI/AReaL/pull/271

Full Changelog: https://github.com/inclusionAI/AReaL/compare/v0.3.0-lite.post2...v0.3.1