Download Latest Version NVIDIA Megatron Core 0.19.2 source code.zip (16.4 MB) Google Add to Preferred Sources
Home / core_v0.19.0
Name Modified Size InfoDownloads / Week
Parent folder
NVIDIA Megatron Core 0.19.0 source code.tar.gz 2026-08-19 14.4 MB
0
NVIDIA Megatron Core 0.19.0 source code.zip 2026-08-19 16.4 MB
2 2 weekly downloads
README.md 2026-08-19 60.0 kB
0
Totals: 3 Items   30.8 MB 2
Changelog Details - Add separate mtp_grad_scale_func for MTP loss scaling by @yfw :: PR: [#3459] - [training migration] Migrate GPT builder by @maanug-nv :: PR: [#4741] - Make Mamba conv params direct mixer params by @wujingyue :: PR: [#4899] - Update oncall reviewer assignment by @Phlip79 :: PR: [#5093] - Pass explicit process groups to hybrid logging by @yashaswikarnati :: PR: [#4781] - Clean up top-level repository files by @Phlip79 :: PR: [#5097] - [main] fix(moe): Fix several bugs for DSA rope and spec. by @yuzhongw-nvidia :: PR: [#3026] - Move MIMO unit tests into models/mimo by @yashaswikarnati :: PR: [#5063] - test: update DeepSeek FSDP2 GB200 memory golden by @wujingyue :: PR: [#5094] - ci: make CI resilient to pip/uv network timeouts by @ko3n1g :: PR: [#5118] - ci: treat docker container-removal conflict as flaky by @ko3n1g :: PR: [#5120] - Fix GDN DTensor splitting for FSDP checkpointing by @conver334 :: PR: [#4843] - Remove DeepEP hardware limit check by @janEbert :: PR: [#4846] - Update transformer-engine dependency to revision 4220403 by @balasaajay :: PR: [#5112] - Fix MoE aux_loss / z_loss gradient scaling with TP > 1 by @deepakn94 :: PR: [#5047] - Update Claude copy workflow to enforce user restrictions and improve error messages by @balasaajay :: PR: [#5117] - Add advisory process group guidance to Claude reviews by @yashaswikarnati :: PR: [#5111] - build: cap pydantic<2.14 in transformer-engine dependency metadata by @cuichenx :: PR: [#5125] - fix(test): skip scalar-less tensorboard event files in resume checks by @ko3n1g :: PR: [#5121] - docs: fix contributor guide typo by @LeSingh1 :: PR: [#4858] - ci(unit-tests): split slow unit-test buckets over 15min SLA by @ko3n1g :: PR: [#5133] - Update MINOR version to 19 by @balasaajay :: PR: [#5096] - Fix Muon QKV split for gated attention by @Moozy23232 :: PR: [#4728] - Fix DSA indexer loss not averaged across micro-batches by @kaimo455 :: PR: [#4070] - Roll input IDs for MTP labels by @yfw :: PR: [#3457] - Adding blackwell tests by @shanmugamr1992 :: PR: [#5113] - Refactor: Move paged stashing Triton kernels by @Phlip79 :: PR: [#5003] - Relax atol for test_router_gating_linear router_dtype=torch.float32 by @adityasingh2400 :: PR: [#4915] - Fix incorrect inference metadata tensor dtypes by @santhnm2 :: PR: [#4855] - Disable TE cross entropy loss fusion by @mchrzanowski :: PR: [#5115] - fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather by @ko3n1g :: PR: [#4982] - Pass TP group to unfused cross entropy by @yashaswikarnati :: PR: [#5128] - test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev by @ko3n1g :: PR: [#5156] - ci(notify): mention mcore-oncall and philipp on critical CI events by @ko3n1g :: PR: [#5152] - Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill by @mathemakitten :: PR: [#3509] - ci: Disable a few gb200 test cases to support 2 branches. by @balasaajay :: PR: [#5151] - build: Switch DSv3 on H100 to HybridEP by @balasaajay :: PR: [#5164] - Add MTP acceptance rate metrics by @yfw :: PR: [#3458] - Nemotron Ultra config for ModelOpt examples by @jenchen13 :: PR: [#5159] - Make MTP / prefix cache stats persist for engine lifetime by @santhnm2 :: PR: [#4101] - Restore Greptile configuration by @Phlip79 :: PR: [#5166] - chore: bump `_code_freeze` workflow to `v1.4.2` by @ko3n1g :: PR: [#5132] - ci: Remove docs build test in favor of release test by @chtruong814 :: PR: [#5182] - Move TE cross entropy guard to training args by @yaoyu-33 :: PR: [#5162] - Fix error in deepseek parser by @tdene :: PR: [#5136] - Clamp `max_new_tokens` in MInf to mirror vllm by @tdene :: PR: [#5181] - Fix logprob slicing for 0 generated token case by @santhnm2 :: PR: [#5167] - [Perf] Fold frozen linear dgrad matmul by @cuichenx :: PR: [#5092] - build: add managed = true to [tool.uv] by @kajalj22 :: PR: [#5190] - Stabilize GB200 inference perf tests against cold-start noise by @shanmugamr1992 :: PR: [#5171] - nvidia style guide audit for getting started folder by @megnvidia :: PR: [#5168] - AI aided audit for Nvidia Style guidance by @megnvidia :: PR: [#5141] - Avoid stat syscall in rerun result validation by @dimapihtar :: PR: [#5107] - fix(combined-1f1b): release loss-node input storage after combined backward by @Wohox :: PR: [#4909] - Enable selective recompute for `norm_out` in GDN layers by @xuantengh :: PR: [#4715] - fix(elastification): align with get_batch + utils refactors by @balasaajay :: PR: [#5194] - docs: Update Latest News in README.md by @sbhavani :: PR: [#3790] - Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. by @cspades :: PR: [#4802] - ci: add smoke tests by @balasaajay :: PR: [#5143] - Add mtp_detach_heads config to detach MTP head inputs by @yfw :: PR: [#3456] - docs: fix install guide NGC container anchor by @Connor-XY :: PR: [#5224] - Fuse per-sequence AlltoAll into a unified one in GDN forward by @xuantengh :: PR: [#4913] - Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path by @yashaswikarnati :: PR: [#5150] - Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool by @wujingyue :: PR: [#4810] - Add named layouts to HyperCommGrid for heterogeneous parallelism by @yashaswikarnati :: PR: [#5148] - Fix wgrad race condition when using double buffers. by @cspades :: PR: [#5222] - Move uneven DTensor distributed fixture to conftest by @wujingyue :: PR: [#5237] - Route bridge communicator cross-grid P2P through a dedicated process group by @yashaswikarnati :: PR: [#5234] - Fix test_split_tensor_along_last_dim to actually assert correctness by @lichenlu :: PR: [#4710] - Add optional group= to common_utils model/data-parallel reduction helpers by @yashaswikarnati :: PR: [#5251] - ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide by @chtruong814 :: PR: [#5278] - Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) by @yashaswikarnati :: PR: [#5260] - Remove checkpoint-time GPU cache reclaim workaround by @shurkat-nvidia :: PR: [#5170] - Remove duplicate nccl_allocator import by @returnL :: PR: [#5057] - fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) by @Connor-XY :: PR: [#5280] - Fix Dockerfile warnings by @janEbert :: PR: [#4856] - Fix fused MLA delayed weight grad hooks by @sraman-rgb :: PR: [#5273] - ci: limit retries on unsuccessful test launches by @balasaajay :: PR: [#5275] - Thread pg_collection into get_model DDP bucket sizing by @yashaswikarnati :: PR: [#5250] - Enable non-deterministic results in model configuration for nemotron tests by @balasaajay :: PR: [#5239] - Stabilize hybrid nanov3 gb200 perf by @shanmugamr1992 :: PR: [#5295] - Clip mtp grads separately when mtp_detach_heads=True by @yfw :: PR: [#4116] - Thread pg_collection into train_step reductions by @yashaswikarnati :: PR: [#5259] - Offline Logits-Based Knowledge Distillation by @AAnoosheh :: PR: [#5019] - Handle None values in sampling parameters by @tdene :: PR: [#5300] - Allow for pre-bound socket to be passed in server by @tdene :: PR: [#5301] - Add moe loss normalization for RL SFT by @pthombre :: PR: [#3956] - Add code owners for optimizer-related files by @janEbert :: PR: [#5297] - Fix EP=1 inference by allocating buffers anyway by @mathemakitten :: PR: [#5233] - Fix crash due to tool call at sequence length by @tdene :: PR: [#5302] - Inference: Cudagraph-aware admission gating in prefill scheduler by @mathemakitten :: PR: [#4870] - Account for reasoning token stripping by @tdene :: PR: [#5313] - Thread pg_collection through wrap_model_chunks_with_ddp by @yashaswikarnati :: PR: [#5328] - Fix LatentMoE theoretical memory estimate by @Wong4j :: PR: [#5145] - Add zstandard package to Docker LTS requirements. Fix nightly failures by @balasaajay :: PR: [#5347] - Thread MIMO support through the stock training loop (schedule + optimizer) by @yashaswikarnati :: PR: [#5333] - ci: default functional test time limit to 4h for release/weekly scopes by @ko3n1g :: PR: [#5360] - Fix memory leak with log_max_attention_logit (#4699) by @asolergi-nv :: PR: [#5067] - Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals by @ilml :: PR: [#5351] - Add full model cuda graph support for MTP inference by @santhnm2 :: PR: [#4950] - Expand the Mamba prefix caching memory safety check to include scratch space buffers by @santhnm2 :: PR: [#5348] - Make Megatron RL only materialize last token logit by @tdene :: PR: [#4551] - Profiling by @jalbericiola :: PR: [#3110] - Support fused MLA QKV checkpoint reload by @sraman-rgb :: PR: [#5310] - Add minimal DBuffer implementation by @wujingyue :: PR: [#4835] - [split 1/4] Fix packed THD RoPE under CP by @HollowMan6 :: PR: [#5243] - Document agent PR commit sign-off and signing by @wujingyue :: PR: [#5381] - Remove unused distributed pytest markers by @wujingyue :: PR: [#5380] - [feat] Support fine-grained activation offloading in fused group mlp by @lhb8125 :: PR: [#5082] - Thread tensor-parallel group into the RADIO patch embedder by @yashaswikarnati :: PR: [#5371] - Add MimoModel.zero_grad_buffer delegating to active DDP submodules by @yashaswikarnati :: PR: [#5372] - [split 3/4] Refactor absorbed MLA projection handling by @HollowMan6 :: PR: [#5245] - ci: Remove sync skills workflow by @chtruong814 :: PR: [#5091] - Add flaky marker to fine-grained activation offloading test (#5350) by @balasaajay :: PR: [#5368] - Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" by @balasaajay :: PR: [#5366] - Update goldens for weekly tests after pytorch and TE bumps. by @balasaajay :: PR: [#5399] - Add MIMO runtime setup: per-role RNG seeding and DDP wrapping by @yashaswikarnati :: PR: [#5285] - Add --mamba-training-ssm-states-dtype argument by @tdene :: PR: [#5309] - Fix Mamba prefix match for chunked prefill by @lmcafee-nvidia :: PR: [#4758] - Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes by @wdykas :: PR: [#5187] - Disag MR1: Add inference shard specs and pg-collection building by @wdykas :: PR: [#5186] - Support the MIMO cross-grid path in training loop by @yashaswikarnati :: PR: [#5373] - Stabilize hybrid_2b GB200 perf test against run-to-run noise by @shanmugamr1992 :: PR: [#5364] - Consistent oncall schedule by @Phlip79 :: PR: [#5404] - Disag MR3: Add heterogeneous KV/Mamba reshard planners by @wdykas :: PR: [#5188] - Add RADIO vision encoder wrapper for MIMO example by @yashaswikarnati :: PR: [#5397] - Clean up MTP inference control flow by @santhnm2 :: PR: [#5418] - Add MIMO dual gradient finalization (colocated + non-colocated) by @yashaswikarnati :: PR: [#5286] - Add RL rollout submission and consumption granularity controls by @lauradang :: PR: [#5306] - Add --functional-test-name to trigger_internal_ci by @ko3n1g :: PR: [#5449] - Rename CP batch helpers to describe balancing granularity by @deepakn94 :: PR: [#5403] - build: point flash_mla at the nv_dev branch by @ko3n1g :: PR: [#5448] - Add logprobs_mode (raw/processed) to inference config by @tdene :: PR: [#5419] - ci: Set test_save_verify_integrity_manifest_directly as flaky by @chtruong814 :: PR: [#5468] - Remove DBuffer mesh axis validation by @wujingyue :: PR: [#5441] - Add hetero grid args and MoE process groups for MIMO example by @yashaswikarnati :: PR: [#5375] - Support SWA and sink attention in dynamic inference by @cuichenx :: PR: [#5249] - feat(inference): default use_coordinator to True in high-level APIs by @shanmugamr1992 :: PR: [#5326] - Support HybridModel feature specs in ModelOpt by @Phlip79 :: PR: [#5354] - Add experimental Megatron-FSDP fully_shard implementation by @wujingyue :: PR: [#5387] - Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions by @shanmugamr1992 :: PR: [#5169] - ci: launch GB200 unit tests via launch_on_gb200 marker by @ko3n1g :: PR: [#5477] - build: install flash_mla from source in the CI image by @ko3n1g :: PR: [#5481] - [split 2/4] Scale DSA indexer loss in pipeline schedules by @HollowMan6 :: PR: [#5244] - ci: check megatron.training imports in installation test by @ko3n1g :: PR: [#5458] - Fix merges_file kwarg name in HuggingFaceTokenizer by @muyihao :: PR: [#5406] - Clean up training.py module header (dedupe + reorganize imports/globals) by @ilml :: PR: [#5469] - Automated community request assignment by @Phlip79 :: PR: [#5147] - Thread process groups through training checkpoint paths by @yashaswikarnati :: PR: [#5486] - Narrow oncall responsibilities by @Phlip79 :: PR: [#5490] - Add MIMO forward step and per-token loss for hetero training by @yashaswikarnati :: PR: [#5376] - Add Nemotron6-MoE VLM model provider for MIMO example by @yashaswikarnati :: PR: [#5374] - ci: auto-retry test-data download in container-build job by @ko3n1g :: PR: [#5498] - Force RL inference to CP=1 by @tdene :: PR: [#5423] - Merge cu_seqlens across micro-batch for THD attention by @deepakn94 :: PR: [#5454] - [split 4/4] Enable DSA CP and THD hooks by @HollowMan6 :: PR: [#5246] - Fix fused MLA down projection with tensor parallelism by @sraman-rgb :: PR: [#5383] - Fix NameError in is_flashinfer_min_version when check_equality=False by @adityasingh2400 :: PR: [#4961] - Add hybrid FSDP unit module support by @Phlip79 :: PR: [#4329] - fix: set DATA_PATH for moe-dynamic-inference recipe by @ko3n1g :: PR: [#5506] - Add --qad-train-target {base|mtp|both} for QAD / MTP QAT (frozen-base, frozen-MTP, or co-train) by @yeyu-nvidia :: PR: [#4785] - [Main] Generalized fix for mxfp8 param gather by @zhongbozhu :: PR: [#5236] - test: restore G/G + lag=19 for gpt_grpo_tp4_pp1_dp2_8b throughput tests by @lauradang :: PR: [#5514] - ci: cache-from a single coherent buildcache donor by @ko3n1g :: PR: [#5509] - Add CUDA graph training iteration test by @wujingyue :: PR: [#5417] - ci: Use GB300 for Github CI tests by @chtruong814 :: PR: [#5520] - ci: pin HF_HUB_CACHE to bind-mounted cache for gpt-oss-20b inference test by @ko3n1g :: PR: [#5512] - Add inter-document attention masking to GPTDataset by @deepakn94 :: PR: [#5298] - [CI] Fix `gpt_dynamic_inference_tp2_pp2_ep2_gptoss_20b_swa` tests by @asolergi-nv :: PR: [#5527] - Implement async scheduling for dynamic inference by @lmcafee-nvidia :: PR: [#5453] - build: bump transformer-engine to release_v2.16.post by @ko3n1g :: PR: [#5517] - Fix `isort` target Python version by @janEbert :: PR: [#5567] - Fix PR template typo by @janEbert :: PR: [#5566] - Deduplicate tensor-splitting utility by @anlthms :: PR: [#5545] - Add CI duties to oncall by @Phlip79 :: PR: [#5510] - Thread dp_cp/expt_dp process groups through checkpoint load path by @yashaswikarnati :: PR: [#5579] - Fix TEGroupedMLP pre-backward unshard in fine-grained FSDP hooks for … by @rapatel :: PR: [#4990] - [training migration] Finish ModelBuilder integration by @maanug-nv :: PR: [#5516] - Use NVIDIA inference credentials for Claude actions by @Phlip79 :: PR: [#5589] - Update PR instructions by @Phlip79 :: PR: [#5592] - Update mcore skill owners by @Phlip79 :: PR: [#5586] - Add /claude fix workflow for on-demand PR fixes by @Phlip79 :: PR: [#4862] - [Megatron-FSDP] MaxPoolAllocator for double-buffering hybrid architectures. by @cspades :: PR: [#5462] - fix(tensor_parallel): _reduce returns unreduced tensor for non-contig… by @Pearblossom-M :: PR: [#5338] - Add Auto Quantize in ModelOpt quantize example by @jenchen13 :: PR: [#4821] - [Main][feat] Support CUDA Graph capture offloading modules by @lhb8125 :: PR: [#3697] - E2E heterogenous non colocated MiMo training by @yashaswikarnati :: PR: [#5602] - Optimize memory usage of partial CUDA graphs by @jiemingz :: PR: [#5451] - Document stacked dependent PR handling in split PR skill by @wujingyue :: PR: [#5496] - Fix Claude reaction permissions by @Phlip79 :: PR: [#5613] - Fix smoke BERT/T5 test failures by @balasaajay :: PR: [#5629] - Add NCCL symmetric-memory staging to experimental FSDP by @wujingyue :: PR: [#5440] - Add smoke test notification functionality and update notify script by @balasaajay :: PR: [#5631] - Update golden value files for GPT-3 weekly by @balasaajay :: PR: [#5459] - Ignore contributor DCO failures in Claude fix by @Phlip79 :: PR: [#5625] - Pre-size the all-gather buffer for inference to max capacity by @santhnm2 :: PR: [#5546] - Scatter embeddings for sequence parallelism in standalone LM forwards by @kevalmorabia97 :: PR: [#5628] - Fix inter-document masking crash and NaNs with TP > 1 and micro_batch_size > 1 by @deepakn94 :: PR: [#5635] - add safe version of numpy.load by @dimapihtar :: PR: [#5500] - Fuse shared expert MLP with grouped GEMM by @sraman-rgb :: PR: [#5604] - MoE routing analysis and metrics capture by @mathemakitten :: PR: [#5220] - Add cspades to oncall rotation by @Phlip79 :: PR: [#5695] - Add microbatch context helper by @wujingyue :: PR: [#5652] - ci: Update test configurations to unify legacy scope names by @balasaajay :: PR: [#5316] - Fix Torch FSDP2 crash: add force_all_reduce kwarg to base finish_grad_sync by @factnn :: PR: [#4953] - Separate mFSDP v2 unit tests by @wujingyue :: PR: [#5640] - test(determinism): add determinism tests by @ZhiyuLi-Nvidia :: PR: [#5041] - deprecate common strategy by @dimapihtar :: PR: [#5160] - ci: revert unify legacy scope names (#5316) by @ko3n1g :: PR: [#5709] - Normalize CRLF in Claude fix commands by @Phlip79 :: PR: [#5712] - Remove some barriers in save_checkpoint_and_time by @shurkat-nvidia :: PR: [#5557] - Add FSDP NVTX annotations by @wujingyue :: PR: [#5704] - NCCL EP support by @YangFei1990 :: PR: [#5129] - Update base image to nvcr.io/nvidia/pytorch:26.06-py3 by @balasaajay :: PR: [#5632] - Refactor RL rollout pipeline by @lauradang :: PR: [#5491] - [2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) by @HollowMan6 :: PR: [#5099] - Triton kernels - avoid recompilation and autotuning in prod by @sidsingh-nvidia :: PR: [#5608] - Add NeMo Transformer audio encoder model by @yqwangustc :: PR: [#5565] - remove deprecated modules from core/dist_checkpointing by @dimapihtar :: PR: [#5134] - fix(fsdp): import os in safe_get_rank fallback by @fallintoplace :: PR: [#4959] - Add forward all-gather overlap by @wujingyue :: PR: [#5513] - Fix seq_load_balancing loss with inter-document masking and MBS > 1 by @deepakn94 :: PR: [#5696] - Avoid X11 master port default by @guihong-nv :: PR: [#5299] - Fix infinite recursion in abstract tokenizer special-id property aliases by @asadbekXodjayev :: PR: [#5445] - Set Bert TE spec q/k_layernorm to None by @bbuschkaemper :: PR: [#5687] - Set is_first_microbatch when quant_recipe is configured by @yezhengmao1 :: PR: [#5642] - Assign BERT CODEOWNERS to GPT team by @Phlip79 :: PR: [#5746] - Remove use of exec_module by @jon-barker :: PR: [#5744] - Short-circuit condition to avoid copying from GPU memory in `ChainedOptimizer` by @filaretov :: PR: [#5623] - Increase Megatron-FSDP overlap test dim to 8192 for reliable overlap by @wujingyue :: PR: [#5770] - Inference: Add profile endpoints to chat completions. by @sidsingh-nvidia :: PR: [#5611] - Inference: Do not route pad/dummy tokens to any expert by @sidsingh-nvidia :: PR: [#4922] - Inference: Add load aware routing to prefix caching. by @sidsingh-nvidia :: PR: [#5607] - fix(clip_grads): handle empty grads_for_norm in inf-norm and p-norm paths by @Mattral :: PR: [#5530] - test(gpt): AUT-830 mark tp1_pp4_vp1_resume_torch_decoupled_lr flaky on h100 by @svcnemo-autobot :: PR: [#5801] - Inference: Bring chat completions API inline with vllm/official openAI spec by @sidsingh-nvidia :: PR: [#5276] - Various ModelOpt fixes: QAD test for CICD, use model builder config instead of model provider, allow loading teacher ckpt independently of student by @AAnoosheh :: PR: [#4520] - build: Update Transformer Engine to 2.17 by @balasaajay :: PR: [#5680] - Mamba prefix caching fixes by @santhnm2 :: PR: [#5502] - Exercise nested MFSDP CUDA graph capture by @wujingyue :: PR: [#5796] - Implement Quantile Balancing in MoE by @Mellonta :: PR: [#5349] - Unset NCCL overrides for MFSDP v2 tests by @wujingyue :: PR: [#5794] - Fix MegatronFSDP root module hook dispatch by @wujingyue :: PR: [#5808] - Pin cudnn-fe and cuTeDSL version by @balasaajay :: PR: [#5812] - Inference: Add the nemotron_v3 reasoning parser by @sidsingh-nvidia :: PR: [#5634] - Missing moe_router_dtype causes unexpected downcast in ModelOpt example by @jinhangchoi :: PR: [#5810] - chore(tests): AUT-851 move NCCL defaults from run_ci_test.sh to conftest by @svcnemo-autobot :: PR: [#5826] - Avoid FSDP unit terminology in MFSDP v2 by @wujingyue :: PR: [#5793] - [refactor] Common combined-1F1B schedule-plan base (1/4 of [#4798]) by @Connor-XY :: PR: [#4941] - Fix configured norm epsilon in MambaLayer by @shanhaoli :: PR: [#5750] - Set num_splits to 0 for FA4 inference by @santhnm2 :: PR: [#5804] - ci: Enhance nightly/mr/weekly error reporting by @balasaajay :: PR: [#5831] - feat(docker): Add NCCL installation script and install NCCL 2.30.4 by @balasaajay :: PR: [#5815] - IMA fix by making the copy of book keeping buffer to GPU blocking by @shanmugamr1992 :: PR: [#5715] - Add GPTModel to HybridModel migration guide by @Phlip79 :: PR: [#5698] - Pair frozen FSDP backward hooks by @wujingyue :: PR: [#5710] - Clarify NVIDIA email signing guidance by @wujingyue :: PR: [#5699] - Overlap FSDP communication with compute by @wujingyue :: PR: [#5719] - chore(skills): add Regent Open Plugin manifest by @ko3n1g :: PR: [#5840] - chore(skills): remove Open Plugin manifest (superseded) by @ko3n1g :: PR: [#5842] - return prefix cache hits data from the chat completions api by @sidsingh-nvidia :: PR: [#5609] - Refactor data parallel coordinator to enable modular handlers by @santhnm2 :: PR: [#5550] - Pass device IDs to cleanup barrier by @wujingyue :: PR: [#5702] - Fix averaging for MoE z-loss metric tracking by @Marks101 :: PR: [#3199] - test(mfsdp): AUT-881 mark test_overlaps_communication_and_compute flaky by @svcnemo-autobot :: PR: [#5848] - Inference: Extend default cuda-graph coverage to 512 tokens by @sidsingh-nvidia :: PR: [#5797] - Fix for sequence-level aux MoE loss being dependent on batch size by @OlegSudakov :: PR: [#5798] - Allow parameterless FSDP root modules by @wujingyue :: PR: [#5711] - Fix issue where parameter groups with different min/max LRs get overridden at checkpoint load time by @jstjohn :: PR: [#4705] - fix bug where Gemma4 is not working with recompute_granularity = "full" by @warpuv :: PR: [#5324] - Avoid extra MFSDP v2 model-weight sync memcpy by @wujingyue :: PR: [#5834] - [experimental] Add experimental/agent_compose placeholder with preview pointer by @ISEEKYAN :: PR: [#5639] - Stabilize mfsdp_v2 overlap test by enlarging the model by @wujingyue :: PR: [#5846] - Print important dependencies by @gautham-kollu :: PR: [#5814] - Test zero-CTA copy-engine all-gather by @wujingyue :: PR: [#5858] - Route Lion through DistributedOptimizer and support single-moment checkpointing by @deepakn94 :: PR: [#5742] - Fix broken remove_sharded_tensors public API and re-enable its unit test by @huthvincent :: PR: [#5759] - Make the model larger and higher mb size to make reduce flakiness by @gautham-kollu :: PR: [#5816] - Update active oncall to Phlip79 this week by @Phlip79 :: PR: [#5896] - Overlap async scheduling phases by @lmcafee-nvidia :: PR: [#5549] - ci: integrate nemo-ci-triage with linear issues management for gitlab failures by @balasaajay :: PR: [#5881] - Reduce boilerplate around MultiStorageClient feature checks by @Randl :: PR: [#5269] - Add NeMo waveform audio processor (data-side feature extractor) by @yqwangustc :: PR: [#5570] - Support HSDP deferred DP-outer gradient reduction by @Achyuthan-S :: PR: [#5743] - Add fully_shard_optimizer for mixed-precision FSDP by @wujingyue :: PR: [#5411] - Stabilize perf warmup by @Phlip79 :: PR: [#5913] - Add compatibility between training CGs and CP>1 by @tdene :: PR: [#5894] - Log app_finish_time and app_train_loop_finish_time on early-exit path by @aliardaeker :: PR: [#5864] - Test mFSDP v2 overlap with default and symmetric memory by @wujingyue :: PR: [#5859] - fix: Harden Claude GitHub workflows by @chtruong814 :: PR: [#5408] - Reuse profiler helpers in mFSDP v2 symmetric memory tests by @wujingyue :: PR: [#5873] - Refresh BERT H100 golden values by @Phlip79 :: PR: [#5953] - Reduce MimoOptimizer update-success across the world for cross-grid consensus by @yashaswikarnati :: PR: [#5331] - fix(resharding): stabilize NVSHMEM refit copy service by @wdykas :: PR: [#5915] - [Main] Numerical fix for FC2 expert bias scales when using `use_transformer_engine_op_fuser` by @zhongbozhu :: PR: [#5850] - ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow by @balasaajay :: PR: [#5957] - Inference: Do not let prompt tokens return from the engine, unless requested by @sidsingh-nvidia :: PR: [#5918] - [Main] Numerical fix for moe single grouped weight with fp8 fp4 primary weight and grad norm spikes by @zhongbozhu :: PR: [#5487] - fix(ci): AUT-957 support golden checks in merge queue by @svcnemo-autobot :: PR: [#5989] - Fix formatting error in qwen3_30b_a3b config by @jon-barker :: PR: [#5978] - Inference: Reduce mamba scratch space size by an order of magnitude. by @sidsingh-nvidia :: PR: [#5863] - Make LRU prefix caching eviction policy only evict child blocks by @santhnm2 :: PR: [#5822] - Fix TE grouped MLP fused main-grad setup by @Wong4j :: PR: [#5209] - Batch-invariant train/inference logprob parity by @wdykas :: PR: [#5897] - Move FSDP model weight sync to optimizer post-step by @wujingyue :: PR: [#5949] - fix: allow mtp_num_layers=0 with overlap_moe_expert_parallel_comm by @cuichenx :: PR: [#5912] - Port Multi-Latent Attention to `HybridModel` by @janEbert :: PR: [#4452] - docs(skills): clarify container::lts is the older LTS PyTorch base by @ko3n1g :: PR: [#6008] - test(hybrid): AUT-971 quarantine Nemotron QAD functional test by @svcnemo-autobot :: PR: [#6013] - [main] add thd sequence packing dispatcher support for main by @HaochenYuan :: PR: [#5008] - Add encoder prefetch for heterogeneous MIMO training by @yashaswikarnati :: PR: [#5833] - Use explicit process groups for dataloader checkpoints by @yashaswikarnati :: PR: [#5988] - Refit: local plan building, node-add cache key, and NIXL backend by @wdykas :: PR: [#5872] - [2/2] Add TileLang fused DSA kernels support with THD and CP & Clean up by @HollowMan6 :: PR: [#5049] - Fix FSDP2 SwiGLU checkpointing. by @cspades :: PR: [#5714] - Prevent FlashInfer sampling from running with CUDA graphs by @santhnm2 :: PR: [#5791] - rl: release G-submission gate slots on consumption instead of assembly by @lauradang :: PR: [#5853] - Enable DDP communication overlap for MIMO training by @yashaswikarnati :: PR: [#5979] - [Inference] Set different random seeds for each DP rank for generation. by @cspades :: PR: [#5983] - Inference: Optimized triton kernels to extract mamba states in prefix caching by @sidsingh-nvidia :: PR: [#5866] - Ensure Mamba prefix cache snapshots are recorded for multi-chunk prompts by @santhnm2 :: PR: [#5952] - ci(actions): AUT-977 retry transient log artifact uploads by @svcnemo-autobot :: PR: [#6027] - fix: add additional error checks for flaky failures by @balasaajay :: PR: [#6029] - fix(inference): AUT-980 disable fp8 primary weights in graph tests by @svcnemo-autobot :: PR: [#6038] - chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade by @svcnemo-autobot :: PR: [#5997] - [feat] Generalized Tensor Parallelism (GTP) by @fanshiqing :: PR: [#4967] - Fix gradient-norm undercounting when using EP and TP by @philipcmonk :: PR: [#5916] - fix(inference): MCORE-536 report dropped prompt token lengths by @svcnemo-autobot :: PR: [#6051] - [GTP][Feat] Add one-block-ahead prefetch for GTP grouped-expert weights by @fanshiqing :: PR: [#6057] - chore(codeowners): AUT-1094 add GTP owners by @svcnemo-autobot :: PR: [#6062] - Deprecate GPTModel in favor of HybridModel by @Phlip79 :: PR: [#5911] - fix(dist-ckpt): AUT-1100 restore default strategy factories by @svcnemo-autobot :: PR: [#6065] - Fix CUDA graph correctness issues due to memory bugs by @jiemingz :: PR: [#5975] - refactor: extract and split common logic between GDN & GDN2 by @xuantengh :: PR: [#5843] - dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 loads with --fp8-param-gather by @asolergi-nv :: PR: [#4451] - Add load-time GPT-to-Hybrid checkpoint translation (#5675) by @guihong-nv :: PR: [#5792] - Populate dp process group in auto-built ProcessGroupCollection in pipeline schedules by @ilml :: PR: [#5901] - fix(cuda-graphs): MB-928 align DDP initialization with capture stream by @svcnemo-autobot :: PR: [#6021] - Enforce that the number of optimizer shards used in layout computation is the same used during the training iteration by @deepakn94 :: PR: [#6048] - NCCL EP zero copy by @YangFei1990 :: PR: [#5735] - Optimize unit metadata for fused shared experts by @sraman-rgb :: PR: [#6053] - build: AUT-1117 serialize uv dependency installation by @svcnemo-autobot :: PR: [#6090] - Extend dynamic inference asynchronous scheduling support by @lmcafee-nvidia :: PR: [#5939] - Add support for non-Gym multi-turn environments by @tdene :: PR: [#5312] - Correct prefix-caching ref-count accounting by @tdene :: PR: [#6047] - send pg group for distributed checkpoint validation by @wdykas :: PR: [#6092] - cp: `chore: update pytest version to 9.1.1 in pyproject.toml and uv.lock (6170)` into `core_r0.19.0` by @ko3n1g :: PR: [#6181] - cp: `deps: Update black dependency to version 26.3.0 (6180)` into `core_r0.19.0` by @ko3n1g :: PR: [#6189] - cp: `deps: Update urllib3 to version 2.7.0 (6182)` into `core_r0.19.0` by @ko3n1g :: PR: [#6188] - cp: `test(optimizer): MCORE-560 cover MoE gradient zero counts (6050)` into `core_r0.19.0` by @ko3n1g :: PR: [#6126] - cp: `test(inference): MCORE-561 trust FP8 metadata in DeepSeek checkpoints (6049)` into `core_r0.19.0` by @ko3n1g :: PR: [#6138] - chore: Update transformer-engine version to 2.18.0+cfda3834 by @balasaajay :: PR: [#6220] - Update Nemotron 3 Super GB200 release config by @Phlip79 :: PR: [#6239] - cp: Update DEEPEP_COMMIT in Dockerfile.ci.dev to de0dd1185142c727b9c118a by @svcnemo-autobot :: PR: [#6281] - build: AUT-1353 bump transformer-engine to 2.17.1 by @svcnemo-autobot :: PR: [#6291] - cp: `set weights_only=True (6150)` into `core_r0.19.0` by @ko3n1g :: PR: [#6351] - docs: Update docs version for 0.19.0 by @chtruong814 :: PR: [#6537] - cp: `Update release notes page (6520)` into `core_r0.19.0` by @Phlip79 :: PR: [#6592] - beep boop 🤖: Bumping Megatron Core to v0.19.1 by @nvidia-megatron-lm-release-bot[bot] :: PR: [#6670] - Add separate mtp_grad_scale_func for MTP loss scaling (#3459) by @yfw - [training migration] Migrate GPT builder (#4741) by @maanug-nv - Make Mamba conv params direct mixer params (#4899) by @wujingyue - Update oncall reviewer assignment (#5093) by @Phlip79 - Pass explicit process groups to hybrid logging (#4781) by @yashaswikarnati - Clean up top-level repository files (#5097) by @Phlip79 - [main] fix(moe): Fix several bugs for DSA rope and spec. (#3026) by @yuzhongw-nvidia - Move MIMO unit tests into models/mimo (#5063) by @yashaswikarnati - test: update DeepSeek FSDP2 GB200 memory golden (#5094) by @wujingyue - Fix GDN DTensor splitting for FSDP checkpointing (#4843) by @conver334 - Remove DeepEP hardware limit check (#4846) by @janEbert - Update transformer-engine dependency to revision 4220403 (#5112) by @balasaajay - ci: make CI resilient to pip/uv network timeouts (#5118) by @ko3n1g - ci: treat docker container-removal conflict as flaky (#5120) by @ko3n1g - Fix MoE aux_loss / z_loss gradient scaling with TP > 1 (#5047) by @deepakn94 - Update Claude copy workflow to enforce user restrictions and improve error messages (#5117) by @balasaajay - Add advisory process group guidance to Claude reviews (#5111) by @yashaswikarnati - build: cap pydantic<2.14 in transformer-engine dependency metadata (#5125) by @cuichenx - fix(test): skip scalar-less tensorboard event files in resume checks (#5121) by @ko3n1g - docs: fix contributor guide typo (#4858) by @LeSingh1 - ci(unit-tests): split slow unit-test buckets over 15min SLA (#5133) by @ko3n1g - Fix Muon QKV split for gated attention (#4728) by @Moozy23232 - Fix DSA indexer loss not averaged across micro-batches (#4070) by @kaimo455 - Update MINOR version to 19 (#5096) by @balasaajay - Roll input IDs for MTP labels (#3457) by @yfw - Refactor: Move paged stashing Triton kernels (#5003) by @Phlip79 - Adding blackwell tests (#5113) by @shanmugamr1992 - Relax atol for test_router_gating_linear router_dtype=torch.float32 (#4915) by @adityasingh2400 - Fix incorrect inference metadata tensor dtypes (#4855) by @santhnm2 - Disable TE cross entropy loss fusion (#5115) by @mchrzanowski - Pass TP group to unfused cross entropy (#5128) by @yashaswikarnati - fix(optimizer): gate ChainedOptimizer MXFP8 defer-sync on DDP-level overlap_param_gather (#4982) by @ko3n1g - test(elastification): quarantine flaky test_gumbel_determinism as flaky_in_dev (#5156) by @ko3n1g - ci(notify): mention mcore-oncall and philipp on critical CI events (#5152) by @ko3n1g - Change the cudagraph distribution from linearly to exponentially-decreasing + grid for mixed prefill (#3509) by @mathemakitten - ci: Disable a few gb200 test cases to support 2 branches. (#5151) by @balasaajay - Add MTP acceptance rate metrics (#3458) by @yfw - build: Switch DSv3 on H100 to HybridEP (#5164) by @balasaajay - Nemotron Ultra config for ModelOpt examples (#5159) by @jenchen13 - Make MTP / prefix cache stats persist for engine lifetime (#4101) by @santhnm2 - Restore Greptile configuration (#5166) by @Phlip79 - chore: bump `_code_freeze` workflow to `v1.4.2` (#5132) by @ko3n1g - ci: Remove docs build test in favor of release test (#5182) by @chtruong814 - Move TE cross entropy guard to training args (#5162) by @yaoyu-33 - Fix error in deepseek parser (#5136) by @tdene - Fix logprob slicing for 0 generated token case (#5167) by @santhnm2 - [Perf] Fold frozen linear dgrad matmul (#5092) by @cuichenx - Clamp `max_new_tokens` in MInf to mirror vllm (#5181) by @tdene - build: add managed = true to [tool.uv] (#5190) by @kajalj22 - Stabilize GB200 inference perf tests against cold-start noise (#5171) by @shanmugamr1992 - nvidia style guide audit for getting started folder (#5168) by @megnvidia - AI aided audit for Nvidia Style guidance (#5141) by @megnvidia - Avoid stat syscall in rerun result validation (#5107) by @dimapihtar - Enable selective recompute for `norm_out` in GDN layers (#4715) by @xuantengh - fix(elastification): align with get_batch + utils refactors (#5194) by @balasaajay - fix(combined-1f1b): release loss-node input storage after combined backward (#4909) by @Wohox - chore(beep boop 🤖): Bump (main) (2026-06-08) by @github-actions[bot] - docs: Update Latest News in README.md (#3790) by @sbhavani - Fix bug with Megatron-FSDP zero counter not working with decoupled gradients. (#4802) by @cspades - ci: add smoke tests (#5143) by @balasaajay - Add mtp_detach_heads config to detach MTP head inputs (#3456) by @yfw - docs: fix install guide NGC container anchor (#5224) by @Connor-XY - Fuse per-sequence AlltoAll into a unified one in GDN forward (#4913) by @xuantengh - Apply MIMO SP/CP sharding with explicit groups and enable THD in non-colocated path (#5150) by @yashaswikarnati - Fix CUDA IMA in fsdp_double_buffer when an FSDP unit's bucket doesn't fit the pool (#4810) by @wujingyue - Add named layouts to HyperCommGrid for heterogeneous parallelism (#5148) by @yashaswikarnati - Fix wgrad race condition when using double buffers. (#5222) by @cspades - Move uneven DTensor distributed fixture to conftest (#5237) by @wujingyue - Route bridge communicator cross-grid P2P through a dedicated process group (#5234) by @yashaswikarnati - Fix test_split_tensor_along_last_dim to actually assert correctness (#4710) by @lichenlu - Add optional group= to common_utils model/data-parallel reduction helpers (#5251) by @yashaswikarnati - Add MIMO hetero topology + distributed bootstrap (examples/mimo training-loop folder) (#5260) by @yashaswikarnati - ci: Allow DCO check in merge queue and add DCO requirement to Contribution guide (#5278) by @chtruong814 - Remove checkpoint-time GPU cache reclaim workaround (#5170) by @shurkat-nvidia - Remove duplicate nccl_allocator import (#5057) by @returnL - fix(ci): resolve t5 dataloader stall + GRPO cudagraph-memory regression (CI-validated) (#5280) by @Connor-XY - Fix Dockerfile warnings (#4856) by @janEbert - Fix fused MLA delayed weight grad hooks (#5273) by @sraman-rgb - ci: limit retries on unsuccessful test launches (#5275) by @balasaajay - Thread pg_collection into get_model DDP bucket sizing (#5250) by @yashaswikarnati - Enable non-deterministic results in model configuration for nemotron tests (#5239) by @balasaajay - Stabilize hybrid nanov3 gb200 perf (#5295) by @shanmugamr1992 - Clip mtp grads separately when mtp_detach_heads=True (#4116) by @yfw - Thread pg_collection into train_step reductions (#5259) by @yashaswikarnati - Handle None values in sampling parameters (#5300) by @tdene - Allow for pre-bound socket to be passed in server (#5301) by @tdene - Offline Logits-Based Knowledge Distillation (#5019) by @AAnoosheh - Add moe loss normalization for RL SFT (#3956) by @pthombre - Add code owners for optimizer-related files (#5297) by @janEbert - Fix EP=1 inference by allocating buffers anyway (#5233) by @mathemakitten - Fix crash due to tool call at sequence length (#5302) by @tdene - Inference: Cudagraph-aware admission gating in prefill scheduler (#4870) by @mathemakitten - Account for reasoning token stripping (#5313) by @tdene - Thread pg_collection through wrap_model_chunks_with_ddp (#5328) by @yashaswikarnati - chore(beep boop 🤖): Bump (main) (2026-06-15) by @github-actions[bot] - Fix LatentMoE theoretical memory estimate (#5145) by @Wong4j - Add zstandard package to Docker LTS requirements. Fix nightly failures (#5347) by @balasaajay - Thread MIMO support through the stock training loop (schedule + optimizer) (#5333) by @yashaswikarnati - ci: default functional test time limit to 4h for release/weekly scopes (#5360) by @ko3n1g - Fix memory leak with log_max_attention_logit (#4699) (#5067) by @asolergi-nv - Clean up pretrain_gpt.py and pretrain_hybrid.py formatting and remove module globals (#5351) by @ilml - Add full model cuda graph support for MTP inference (#4950) by @santhnm2 - Expand the Mamba prefix caching memory safety check to include scratch space buffers (#5348) by @santhnm2 - Make Megatron RL only materialize last token logit (#4551) by @tdene - Profiling (#3110) by @jalbericiola - Support fused MLA QKV checkpoint reload (#5310) by @sraman-rgb - Document agent PR commit sign-off and signing (#5381) by @wujingyue - Add minimal DBuffer implementation (#4835) by @wujingyue - [split 1/5] Fix packed THD RoPE under CP (#5243) by @HollowMan6 - Remove unused distributed pytest markers (#5380) by @wujingyue - [feat] Support fine-grained activation offloading in fused group mlp (#5082) by @lhb8125 - Thread tensor-parallel group into the RADIO patch embedder (#5371) by @yashaswikarnati - Add MimoModel.zero_grad_buffer delegating to active DDP submodules (#5372) by @yashaswikarnati - [split 3/5] Refactor absorbed MLA projection handling (#5245) by @HollowMan6 - ci: Remove sync skills workflow (#5091) by @chtruong814 - Add flaky marker to fine-grained activation offloading test (#5350) (#5368) by @balasaajay - Revert "Remove checkpoint-time GPU cache reclaim workaround (#5170)" (#5366) by @balasaajay - Update goldens for weekly tests after pytorch and TE bumps. (#5399) by @balasaajay - Add MIMO runtime setup: per-role RNG seeding and DDP wrapping (#5285) by @yashaswikarnati - Add --mamba-training-ssm-states-dtype argument (#5309) by @tdene - chore(beep boop 🤖): Bump (main) (2026-06-22) by @github-actions[bot] - Fix Mamba prefix match for chunked prefill (#4758) by @lmcafee-nvidia - Disag MR2: Refit into multiple destination pools and tied-embedding + UVM fixes (#5187) by @wdykas - Disag MR1: Add inference shard specs and pg-collection building (#5186) by @wdykas - Support the MIMO cross-grid path in training loop (#5373) by @yashaswikarnati - Stabilize hybrid_2b GB200 perf test against run-to-run noise (#5364) by @shanmugamr1992 - Consistent oncall schedule (#5404) by @Phlip79 - Disag MR3: Add heterogeneous KV/Mamba reshard planners (#5188) by @wdykas - Add RADIO vision encoder wrapper for MIMO example (#5397) by @yashaswikarnati - Clean up MTP inference control flow (#5418) by @santhnm2 - Add MIMO dual gradient finalization (colocated + non-colocated) (#5286) by @yashaswikarnati - Add RL rollout submission and consumption granularity controls (#5306) by @lauradang - Add --functional-test-name to trigger_internal_ci (#5449) by @ko3n1g - Rename CP batch helpers to describe balancing granularity (#5403) by @deepakn94 - build: point flash_mla at the nv_dev branch (#5448) by @ko3n1g - Add logprobs_mode (raw/processed) to inference config (#5419) by @tdene - Remove DBuffer mesh axis validation (#5441) by @wujingyue - Support SWA and sink attention in dynamic inference (#5249) by @cuichenx - Add hetero grid args and MoE process groups for MIMO example (#5375) by @yashaswikarnati - ci: Set test_save_verify_integrity_manifest_directly as flaky (#5468) by @chtruong814 - feat(inference): default use_coordinator to True in high-level APIs (#5326) by @shanmugamr1992 - Support HybridModel feature specs in ModelOpt (#5354) by @Phlip79 - Add experimental Megatron-FSDP fully_shard implementation (#5387) by @wujingyue - Add inference functions to support MCore-/MBridge- training refactor and remove legacy modelbuilder functions (#5169) by @shanmugamr1992 - ci: launch GB200 unit tests via launch_on_gb200 marker (#5477) by @ko3n1g - build: install flash_mla from source in the CI image (#5481) by @ko3n1g - [split 2/4] Scale DSA indexer loss in pipeline schedules (#5244) by @HollowMan6 - ci: check megatron.training imports in installation test (#5458) by @ko3n1g - Fix merges_file kwarg name in HuggingFaceTokenizer (#5406) by @muyihao - Automated community request assignment (#5147) by @Phlip79 - Clean up training.py module header (dedupe + reorganize imports/globals) (#5469) by @ilml - Thread process groups through training checkpoint paths (#5486) by @yashaswikarnati - Narrow oncall responsibilities (#5490) by @Phlip79 - Add MIMO forward step and per-token loss for hetero training (#5376) by @yashaswikarnati - Add Nemotron6-MoE VLM model provider for MIMO example (#5374) by @yashaswikarnati - ci: auto-retry test-data download in container-build job (#5498) by @ko3n1g - Force RL inference to CP=1 (#5423) by @tdene - Merge cu_seqlens across micro-batch for THD attention (#5454) by @deepakn94 - [split 4/4] Enable DSA CP and THD hooks (#5246) by @HollowMan6 - Fix fused MLA down projection with tensor parallelism (#5383) by @sraman-rgb - Fix NameError in is_flashinfer_min_version when check_equality=False (#4961) by @adityasingh2400 - Add hybrid FSDP unit module support (#4329) by @Phlip79 - fix: set DATA_PATH for moe-dynamic-inference recipe (#5506) by @ko3n1g - Add --qad-train-target {base|mtp|both} for QAD / MTP QAT (frozen-base, frozen-MTP, or co-train) (#4785) by @yeyu-nvidia - [Main] Generalized fix for mxfp8 param gather (#5236) by @zhongbozhu - ci: cache-from a single coherent buildcache donor (#5509) by @ko3n1g - Add CUDA graph training iteration test (#5417) by @wujingyue - test: restore G/G + lag=19 for gpt_grpo_tp4_pp1_dp2_8b throughput tests (#5514) by @lauradang - ci: Use GB300 for Github CI tests (#5520) by @chtruong814 - ci: pin HF_HUB_CACHE to bind-mounted cache for gpt-oss-20b inference test (#5512) by @ko3n1g - Add inter-document attention masking to GPTDataset (#5298) by @deepakn94 - [CI] Fix `gpt_dynamic_inference_tp2_pp2_ep2_gptoss_20b_swa` tests (#5527) by @asolergi-nv - Implement async scheduling for dynamic inference (#5453) by @lmcafee-nvidia - build: bump transformer-engine to release_v2.16.post (#5517) by @ko3n1g - Fix `isort` target Python version (#5567) by @janEbert - Fix PR template typo (#5566) by @janEbert - Deduplicate tensor-splitting utility (#5545) by @anlthms - Add CI duties to oncall (#5510) by @Phlip79 - Thread dp_cp/expt_dp process groups through checkpoint load path (#5579) by @yashaswikarnati - Fix TEGroupedMLP pre-backward unshard in fine-grained FSDP hooks for … (#4990) by @rapatel - [training migration] Finish ModelBuilder integration (#5516) by @maanug-nv - Use NVIDIA inference credentials for Claude actions (#5589) by @Phlip79 - Update PR instructions (#5592) by @Phlip79 - Update mcore skill owners (#5586) by @Phlip79 - Add /claude fix workflow for on-demand PR fixes (#4862) by @Phlip79 - [Megatron-FSDP] MaxPoolAllocator for double-buffering hybrid architectures. (#5462) by @cspades - fix(tensor_parallel): _reduce returns unreduced tensor for non-contig… (#5338) by @Pearblossom-M - Add Auto Quantize in ModelOpt quantize example (#4821) by @jenchen13 - [Main][feat] Support CUDA Graph capture offloading modules (#3697) by @lhb8125 - E2E heterogenous non colocated MiMo training (#5602) by @yashaswikarnati - Optimize memory usage of partial CUDA graphs (#5451) by @jiemingz - Document stacked dependent PR handling in split PR skill (#5496) by @wujingyue - Fix Claude reaction permissions (#5613) by @Phlip79 - Fix smoke BERT/T5 test failures (#5629) by @balasaajay - Add NCCL symmetric-memory staging to experimental FSDP (#5440) by @wujingyue - Add smoke test notification functionality and update notify script (#5631) by @balasaajay - Update golden value files for GPT-3 weekly (#5459) by @balasaajay - Ignore contributor DCO failures in Claude fix (#5625) by @Phlip79 - chore(beep boop 🤖): Bump (main) (2026-07-06) by @github-actions[bot] - Pre-size the all-gather buffer for inference to max capacity (#5546) by @santhnm2 - Scatter embeddings for sequence parallelism in standalone LM forwards (#5628) by @kevalmorabia97 - Fix inter-document masking crash and NaNs with TP > 1 and micro_batch_size > 1 (#5635) by @deepakn94 - add safe version of numpy.load (#5500) by @dimapihtar - Fuse shared expert MLP with grouped GEMM (#5604) by @sraman-rgb - MoE routing analysis and metrics capture (#5220) by @mathemakitten - Add cspades to oncall rotation (#5695) by @Phlip79 - Add microbatch context helper (#5652) by @wujingyue - ci: Update test configurations to unify legacy scope names (#5316) by @balasaajay - Fix Torch FSDP2 crash: add force_all_reduce kwarg to base finish_grad_sync (#4953) by @factnn - Separate mFSDP v2 unit tests (#5640) by @wujingyue - test(determinism): add determinism tests (#5041) by @ZhiyuLi-Nvidia - deprecate common strategy (#5160) by @dimapihtar - ci: revert unify legacy scope names (#5316) (#5709) by @ko3n1g - Normalize CRLF in Claude fix commands (#5712) by @Phlip79 - Remove some barriers in save_checkpoint_and_time (#5557) by @shurkat-nvidia - Add FSDP NVTX annotations (#5704) by @wujingyue - NCCL EP support (#5129) by @YangFei1990 - Update base image to nvcr.io/nvidia/pytorch:26.06-py3 (#5632) by @balasaajay - Refactor RL rollout pipeline (#5491) by @lauradang - [2/2] Wiring cuDNN fused DSA kernels support with THD, CP and IndexShare (GLM5.2) (#5099) by @HollowMan6 - Triton kernels - avoid recompilation and autotuning in prod (#5608) by @sidsingh-nvidia - Add NeMo Transformer audio encoder model (#5565) by @yqwangustc - remove deprecated modules from core/dist_checkpointing (#5134) by @dimapihtar - fix(fsdp): import os in safe_get_rank fallback (#4959) by @fallintoplace - Add forward all-gather overlap (#5513) by @wujingyue - Fix seq_load_balancing loss with inter-document masking and MBS > 1 (#5696) by @deepakn94 - Avoid X11 master port default (#5299) by @guihong-nv - Fix infinite recursion in abstract tokenizer special-id property aliases (#5445) by @asadbekXodjayev - Set Bert TE spec q/k_layernorm to None (#5687) by @bbuschkaemper - Set is_first_microbatch when quant_recipe is configured (#5642) by @yezhengmao1 - Assign BERT CODEOWNERS to GPT team (#5746) by @Phlip79 - Remove use of exec_module (#5744) by @jon-barker - Short-circuit condition to avoid copying from GPU memory in `ChainedOptimizer` (#5623) by @filaretov - Increase Megatron-FSDP overlap test dim to 8192 for reliable overlap (#5770) by @wujingyue - Inference: Add profile endpoints to chat completions. (#5611) by @sidsingh-nvidia - Inference: Do not route pad/dummy tokens to any expert (#4922) by @sidsingh-nvidia - Inference: Add load aware routing to prefix caching. (#5607) by @sidsingh-nvidia - fix(clip_grads): handle empty grads_for_norm in inf-norm and p-norm paths (#5530) by @Mattral - test(gpt): AUT-830 mark tp1_pp4_vp1_resume_torch_decoupled_lr flaky on h100 (#5801) by @svcnemo-autobot - Delegate reasoning token retention to the chat template in multi-turn conversations (#5276) by @sidsingh-nvidia - Various ModelOpt fixes: QAD test for CICD, use model builder config instead of model provider, allow loading teacher ckpt independently of student (#4520) by @AAnoosheh - build: Update Transformer Engine to 2.17 (#5680) by @balasaajay - Mamba prefix caching fixes (#5502) by @santhnm2 - Exercise nested MFSDP CUDA graph capture (#5796) by @wujingyue - Implement Quantile Balancing in MoE (#5349) by @Mellonta - Unset NCCL overrides for MFSDP v2 tests (#5794) by @wujingyue - Fix MegatronFSDP root module hook dispatch (#5808) by @wujingyue - Pin cudnn-fe and cuTeDSL version (#5812) by @balasaajay - Inference: Add the nemotron_v3 reasoning parser (#5634) by @sidsingh-nvidia - Missing moe_router_dtype causes unexpected downcast in ModelOpt example (#5810) by @jinhangchoi - Avoid FSDP unit terminology in MFSDP v2 (#5793) by @wujingyue - Fix configured norm epsilon in MambaLayer (#5750) by @shanhaoli - [refactor] Common combined-1F1B schedule-plan base (1/4 of [#4798]) (#4941) by @Connor-XY - chore(tests): AUT-851 move NCCL defaults from run_ci_test.sh to conftest (#5826) by @svcnemo-autobot - Set num_splits to 0 for FA4 inference (#5804) by @santhnm2 - ci: Enhance nightly/mr/weekly error reporting (#5831) by @balasaajay - feat(docker): Add NCCL installation script and install NCCL 2.30.4 (#5815) by @balasaajay - IMA fix by making the copy of book keeping buffer to GPU blocking (#5715) by @shanmugamr1992 - Add GPTModel to HybridModel migration guide (#5698) by @Phlip79 - Pair frozen FSDP backward hooks (#5710) by @wujingyue - Clarify NVIDIA email signing guidance (#5699) by @wujingyue - Overlap FSDP communication with compute (#5719) by @wujingyue - chore(skills): add Regent Open Plugin manifest (#5840) by @ko3n1g - return prefix cache hits data from the chat completions api (#5609) by @sidsingh-nvidia - chore(skills): remove Open Plugin manifest (superseded) (#5842) by @ko3n1g - Refactor data parallel coordinator to enable modular handlers (#5550) by @santhnm2 - Pass device IDs to cleanup barrier (#5702) by @wujingyue - Fix averaging for MoE z-loss metric tracking (#3199) by @Marks101 - Inference: Extend default cuda-graph coverage to 512 tokens (#5797) by @sidsingh-nvidia - test(mfsdp): AUT-881 mark test_overlaps_communication_and_compute flaky (#5848) by @svcnemo-autobot - Fix for sequence-level aux MoE loss being dependent on batch size (#5798) by @OlegSudakov - Allow parameterless FSDP root modules (#5711) by @wujingyue - Fix issue where parameter groups with different min/max LRs get overridden at checkpoint load time (#4705) by @jstjohn - fix bug where Gemma4 is not working with recompute_granularity = "full" (#5324) by @warpuv - Avoid extra MFSDP v2 model-weight sync memcpy (#5834) by @wujingyue - [experimental] Add experimental/agent_compose placeholder with preview pointer (#5639) by @ISEEKYAN - Stabilize mfsdp_v2 overlap test by enlarging the model (#5846) by @wujingyue - Print important dependencies (#5814) by @gautham-kollu - Test zero-CTA copy-engine all-gather (#5858) by @wujingyue - Route Lion through DistributedOptimizer and support single-moment checkpointing (#5742) by @deepakn94 - Fix broken remove_sharded_tensors public API and re-enable its unit test (#5759) by @huthvincent - Make the model larger and higher mb size to make reduce flakiness (#5816) by @gautham-kollu - chore(beep boop 🤖): Bump (main) (2026-07-20) by @github-actions[bot] - Update active oncall to Phlip79 this week (#5896) by @Phlip79 - Overlap async scheduling phases (#5549) by @lmcafee-nvidia - ci: integrate nemo-ci-triage with linear issues management for gitlab failures (#5881) by @balasaajay - Reduce boilerplate around MultiStorageClient feature checks (#5269) by @Randl - Add NeMo waveform audio processor (data-side feature extractor) (#5570) by @yqwangustc - Support HSDP deferred DP-outer gradient reduction (#5743) by @Achyuthan-S - Add fully_shard_optimizer for mixed-precision FSDP (#5411) by @wujingyue - Stabilize perf warmup (#5913) by @Phlip79 - Add compatibility between training CGs and CP>1 (#5894) by @tdene - Log app_finish_time and app_train_loop_finish_time on early-exit path (#5864) by @aliardaeker - Test mFSDP v2 overlap with default and symmetric memory (#5859) by @wujingyue - fix: Harden Claude GitHub workflows (#5408) by @chtruong814 - chore: rotate oncall schedule by @github-actions[bot] - Reuse profiler helpers in mFSDP v2 symmetric memory tests (#5873) by @wujingyue - Reduce MimoOptimizer update-success across the world for cross-grid consensus (#5331) by @yashaswikarnati - Refresh BERT H100 golden values (#5953) by @Phlip79 - fix(resharding): stabilize NVSHMEM refit copy service (#5915) by @wdykas - [Main] Numerical fix for FC2 expert bias scales when using `use_transformer_engine_op_fuser` (#5850) by @zhongbozhu - ci: Integrates the latest config-driven nemo-ci-triage Slack and Linear workflow (#5957) by @balasaajay - [Main] Numerical fix for moe single grouped weight with fp8 fp4 primary weight and grad norm spikes (#5487) by @zhongbozhu - Inference: Do not let prompt tokens return from the engine, unless requested (#5918) by @sidsingh-nvidia - Fix formatting error in qwen3_30b_a3b config (#5978) by @jon-barker - Make LRU prefix caching eviction policy only evict child blocks (#5822) by @santhnm2 - Inference: Reduce mamba scratch space size by an order of magnitude. (#5863) by @sidsingh-nvidia - Fix TE grouped MLP fused main-grad setup (#5209) by @Wong4j - fix(ci): AUT-957 support golden checks in merge queue (#5989) by @svcnemo-autobot - Batch-invariant train/inference logprob parity (#5897) by @wdykas - Move FSDP model weight sync to optimizer post-step (#5949) by @wujingyue - fix: allow mtp_num_layers=0 with overlap_moe_expert_parallel_comm (#5912) by @cuichenx - Port Multi-Latent Attention to `HybridModel` (#4452) by @janEbert - [main] add thd sequence packing dispatcher support for main (#5008) by @HaochenYuan - docs(skills): clarify container::lts is the older LTS PyTorch base (#6008) by @ko3n1g - test(hybrid): AUT-971 quarantine Nemotron QAD functional test (#6013) by @svcnemo-autobot - Add encoder prefetch for heterogeneous MIMO training (#5833) by @yashaswikarnati - Use explicit process groups for dataloader checkpoints (#5988) by @yashaswikarnati - [2/2] Add TileLang fused DSA kernels support with THD and CP & Clean up (#5049) by @HollowMan6 - Refit: local plan building, node-add cache key, and NIXL backend (#5872) by @wdykas - Fix FSDP2 SwiGLU checkpointing. (#5714) by @cspades - Prevent FlashInfer sampling from running with CUDA graphs (#5791) by @santhnm2 - Enable DDP communication overlap for MIMO training (#5979) by @yashaswikarnati - rl: release G-submission gate slots on consumption instead of assembly (#5853) by @lauradang - Inference: Optimized triton kernels to extract mamba states in prefix caching (#5866) by @sidsingh-nvidia - [Inference] Set different random seeds for each DP rank for generation. (#5983) by @cspades - Ensure Mamba prefix cache snapshots are recorded for multi-chunk prompts (#5952) by @santhnm2 - ci(actions): AUT-977 retry transient log artifact uploads (#6027) by @svcnemo-autobot - fix: add additional error checks for flaky failures (#6029) by @balasaajay - fix(inference): AUT-980 disable fp8 primary weights in graph tests (#6038) by @svcnemo-autobot - chore(deps): AUT-967 stabilize Transformer Engine 2.18 upgrade (#5997) by @svcnemo-autobot - [feat] Generalized Tensor Parallelism (GTP) (#4967) by @fanshiqing - Fix gradient-norm undercounting when using EP and TP (#5916) by @philipcmonk - fix(inference): MCORE-536 report dropped prompt token lengths (#6051) by @svcnemo-autobot - [GTP][Feat] Add one-block-ahead prefetch for GTP grouped-expert weights (#6057) by @fanshiqing - chore(codeowners): AUT-1094 add GTP owners (#6062) by @svcnemo-autobot - fix(dist-ckpt): AUT-1100 restore default strategy factories (#6065) by @svcnemo-autobot - Deprecate GPTModel in favor of HybridModel (#5911) by @Phlip79 - Fix CUDA graph correctness issues due to memory bugs (#5975) by @jiemingz - Update copy-pr-bot.yaml [skip ci] by @github-actions[bot] - refactor: extract and split common logic between GDN & GDN2 (#5843) by @xuantengh - Add load-time GPT-to-Hybrid checkpoint translation (#5675) (#5792) by @guihong-nv - dist_ckpt: add --stream-ckpt-dequant to fix OOM on large FP8/MXFP8 loads with --fp8-param-gather (#4451) by @asolergi-nv - Populate dp process group in auto-built ProcessGroupCollection in pipeline schedules (#5901) by @ilml - fix(cuda-graphs): MB-928 align DDP initialization with capture stream (#6021) by @svcnemo-autobot - Enforce that the number of optimizer shards used in layout computation is the same used during the training iteration (#6048) by @deepakn94 - NCCL EP zero copy (#5735) by @YangFei1990 - Optimize unit metadata for fused shared experts (#6053) by @sraman-rgb - build: AUT-1117 serialize uv dependency installation (#6090) by @svcnemo-autobot - Extend dynamic inference asynchronous scheduling support (#5939) by @lmcafee-nvidia - Add support for non-Gym multi-turn environments (#5312) by @tdene - Correct prefix-caching ref-count accounting (#6047) by @tdene - send pg group for distributed checkpoint validation (#6092) by @wdykas - cp: `chore: update pytest version to 9.1.1 in pyproject.toml and uv.lock (6170)` into `core_r0.19.0` (#6181) by @ko3n1g - cp: `deps: Update black dependency to version 26.3.0 (6180)` into `core_r0.19.0` (#6189) by @ko3n1g - cp: `deps: Update urllib3 to version 2.7.0 (6182)` into `core_r0.19.0` (#6188) by @ko3n1g - cp: `test(optimizer): MCORE-560 cover MoE gradient zero counts (6050)` into `core_r0.19.0` (#6126) by @ko3n1g - cp: `test(inference): MCORE-561 trust FP8 metadata in DeepSeek checkpoints (6049)` into `core_r0.19.0` (#6138) by @ko3n1g - chore: Update transformer-engine version to 2.18.0+cfda3834 (#6220) by @balasaajay - Update Nemotron 3 Super GB200 release config (#6239) by @Phlip79 - cp: Update DEEPEP_COMMIT in Dockerfile.ci.dev to de0dd1185142c727b9c118a (#6281) by @svcnemo-autobot - build: AUT-1353 bump transformer-engine to 2.17.1 (#6291) by @svcnemo-autobot - cp: `set weights_only=True (6150)` into `core_r0.19.0` (#6351) by @ko3n1g - chore(beep boop 🤖): Bump (core_r0.19.0) (2026-08-10) by @github-actions[bot] - docs: Update docs version for 0.19.0 (#6537) by @chtruong814 - chore(beep boop 🤖): Bump (core_r0.19.0) (2026-08-17) by @github-actions[bot] - cp: `Update release notes page (6520)` into `core_r0.19.0` (#6592) by @Phlip79 - beep boop 🤖: Bumping Megatron Core to v0.19.1 [skip ci] by @github-actions[bot]
Source: README.md, updated 2026-08-19