Olmo-core Files

What's Changed

• Rename scatter_object to broadcast_object() by @epwalsh in https://github.com/allenai/OLMo-core/pull/471
• Add exponential LR scheduler by @baileykuehl in https://github.com/allenai/OLMo-core/pull/472
• Fix typos in code by @agupta2304 in https://github.com/allenai/OLMo-core/pull/473
• Add Peri-LN transformer block, module for residual stream by @epwalsh in https://github.com/allenai/OLMo-core/pull/457
• Fix QK norm parameter count in AttentionConfig.num_params() by @dirkgr in https://github.com/allenai/OLMo-core/pull/477
• Update docker images and versions (most recent image - torch 2.9.1, cuda 13.0) by @tyler-romero in https://github.com/allenai/OLMo-core/pull/459
• Internal olmo3 32b midtrain/long context scripts by @tyler-romero in https://github.com/allenai/OLMo-core/pull/450
• Use correct peak LR in Olmo 3 32B midtrain by @soldni in https://github.com/allenai/OLMo-core/pull/481
• Load multiple checkpoints at the same time and average them by @dirkgr in https://github.com/allenai/OLMo-core/pull/396
• Fix type annotation issue in NumpyInterleavedFSLDataset (Issue [#486]) by @ada-ggf25 in https://github.com/allenai/OLMo-core/pull/487
• Composable data loading by @epwalsh in https://github.com/allenai/OLMo-core/pull/416
• [MoE] Upstream TrainModuleConfig by @tyler-romero in https://github.com/allenai/OLMo-core/pull/490
• various improvements to set up for new model ladder by @epwalsh in https://github.com/allenai/OLMo-core/pull/493
• set num_execution_units default to 1 by @epwalsh in https://github.com/allenai/OLMo-core/pull/495
• split off transformer and attention CPU tests by @epwalsh in https://github.com/allenai/OLMo-core/pull/496
• Changes the prefix used by GPUMemoryMonitorCallback to avoid using the reserved system prefix by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/497
• Avoid torch recompile when intra-doc masking enabled by @epwalsh in https://github.com/allenai/OLMo-core/pull/504
• Gated Attention by @tyler-romero in https://github.com/allenai/OLMo-core/pull/508
• Model ladder, revamped by @epwalsh in https://github.com/allenai/OLMo-core/pull/494
• No-global-rope (GNoPE) by @tyler-romero in https://github.com/allenai/OLMo-core/pull/509
• Fix metrics filter by changing system/ prefix to new gpu_memory/ prefix by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/499
• Add in-loop LM evals to ladder by @epwalsh in https://github.com/allenai/OLMo-core/pull/512
• Bottom-up flops estimation & total flops tracking by @tyler-romero in https://github.com/allenai/OLMo-core/pull/511
• MXFP8 Linear Support by @tyler-romero in https://github.com/allenai/OLMo-core/pull/510
• ladder lmevaluator fix by @tyler-romero in https://github.com/allenai/OLMo-core/pull/514
• Fix overflow when too many global flops are computed by @tyler-romero in https://github.com/allenai/OLMo-core/pull/515
• Jacobm final merge sft by @jacob-morrison in https://github.com/allenai/OLMo-core/pull/400
• Add instance filter to olmo3 ladder by @tyler-romero in https://github.com/allenai/OLMo-core/pull/516
• Make data loading preprocessing more robust to race conditions by @epwalsh in https://github.com/allenai/OLMo-core/pull/517
• Add scripts for slurm env by @epwalsh in https://github.com/allenai/OLMo-core/pull/518
• update slurm scripts by @epwalsh in https://github.com/allenai/OLMo-core/pull/519
• Allow cordoning nodes by @epwalsh in https://github.com/allenai/OLMo-core/pull/520
• More slurm improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/521
• Add temp & mem healthchecks and allow comments in cordoned nodelist by @tyler-romero in https://github.com/allenai/OLMo-core/pull/522
• Consolidate bash helper functions by @epwalsh in https://github.com/allenai/OLMo-core/pull/523
• Increase ulimit -n within slurm jobs by @tyler-romero in https://github.com/allenai/OLMo-core/pull/524
• More Slurm util improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/525
• Ladder improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/529
• GAP Monitor Improvements by @tyler-romero in https://github.com/allenai/OLMo-core/pull/527
• Ulysses CP Support by @tyler-romero in https://github.com/allenai/OLMo-core/pull/532
• Support for Muon and Dion Optimizers by @tyler-romero in https://github.com/allenai/OLMo-core/pull/528
• Add a model ladder for "more norms" and other internal ladder improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/535
• Add 60M and 1M param model configs by @tyler-romero in https://github.com/allenai/OLMo-core/pull/536
• Add Qwen3 model support (0.6B-32B) by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/533
• update to Beaker-py v2, use gantry API internally by @epwalsh in https://github.com/allenai/OLMo-core/pull/540
• Add 60M and 100M param ladder rungs by @tyler-romero in https://github.com/allenai/OLMo-core/pull/543
• always include GOOGLE_CREDENTIALS by @epwalsh in https://github.com/allenai/OLMo-core/pull/545
• Ensure final metrics always logged by @epwalsh in https://github.com/allenai/OLMo-core/pull/546
• Log chinchilla multiple from SpeedMonitorCallback by @epwalsh in https://github.com/allenai/OLMo-core/pull/548
• improve beaker client caching by @epwalsh in https://github.com/allenai/OLMo-core/pull/549
• Add a stepped WSDS variant, integrated into ladder by @epwalsh in https://github.com/allenai/OLMo-core/pull/538
• update base LR for stepped schedule by @epwalsh in https://github.com/allenai/OLMo-core/pull/551
• Add Gemma 3 support with HF weight conversion by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/534
• Add 14M param config by @tyler-romero in https://github.com/allenai/OLMo-core/pull/552
• Fix flaky and broken transformer model tests by @tyler-romero in https://github.com/allenai/OLMo-core/pull/537
• Add model ladders for recently run ablations by @tyler-romero in https://github.com/allenai/OLMo-core/pull/553
• Fix 1M model by @tyler-romero in https://github.com/allenai/OLMo-core/pull/555
• Now, WandBCallback and CometCallback log the Beaker URL to their config sections. by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/547
• Skip init_process_group if already initialized by @finbarrtimbers in https://github.com/allenai/OLMo-core/pull/539
• Fix failing test_build_world_mesh_cpu for pytorch 2.10 by @tyler-romero in https://github.com/allenai/OLMo-core/pull/559
• Easier discovery and collection of ladder results by @tyler-romero in https://github.com/allenai/OLMo-core/pull/556
• Reduce disk usage of convert_checkpoint_to_hf_test by @tyler-romero in https://github.com/allenai/OLMo-core/pull/561
• More ladder work by @epwalsh in https://github.com/allenai/OLMo-core/pull/560
• Add torch 2.10 image, fix FA3/TE incompatibility by @epwalsh in https://github.com/allenai/OLMo-core/pull/562
• Add support for FA4 (CUTE implementation) by @epwalsh in https://github.com/allenai/OLMo-core/pull/563
• Use uv in CI by @epwalsh in https://github.com/allenai/OLMo-core/pull/564
• Add Callback.pre_log_metrics() method by @epwalsh in https://github.com/allenai/OLMo-core/pull/566
• Replace OmegaConf with dataclass-extensions by @epwalsh in https://github.com/allenai/OLMo-core/pull/567
• Utilize Registrable for optim/scheduler configs by @epwalsh in https://github.com/allenai/OLMo-core/pull/568
• Integrate QuACK kernels, minor improvements to Docker build by @epwalsh in https://github.com/allenai/OLMo-core/pull/569
• SequenceMixer as base for both attention and recurrent layers by @tyler-romero in https://github.com/allenai/OLMo-core/pull/570
• update deps by @epwalsh in https://github.com/allenai/OLMo-core/pull/574
• Add GatedDeltaNet by @tyler-romero in https://github.com/allenai/OLMo-core/pull/572
• Add option to load downstream eval tasks lazily by @epwalsh in https://github.com/allenai/OLMo-core/pull/575
• [QoL] include throughput metrics when updating Beaker workload description by @epwalsh in https://github.com/allenai/OLMo-core/pull/577
• Ensure Beaker workloads submitted from CI are canceled gracefully by @epwalsh in https://github.com/allenai/OLMo-core/pull/579
• Evaluate upon completion of training by @tyler-romero in https://github.com/allenai/OLMo-core/pull/580
• Fix documentation link for OLMo-core Beaker images by @natolambert in https://github.com/allenai/OLMo-core/pull/581
• Update SFT documentation with tokenization and training tips by @natolambert in https://github.com/allenai/OLMo-core/pull/571
• Join bookkeeping ops before checkpointing by @epwalsh in https://github.com/allenai/OLMo-core/pull/583
• Propagate gantry launch options by @tyler-romero in https://github.com/allenai/OLMo-core/pull/582
• Fix some logic around soft timeouts in launch.beaker by @epwalsh in https://github.com/allenai/OLMo-core/pull/584
• Make SLACK_WEBHOOK_URL not actually required by @tyler-romero in https://github.com/allenai/OLMo-core/pull/585
• Optional disk-backed caching for remote IO ops, data loading config improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/586
• Add chat template verification step to SFT README by @natolambert in https://github.com/allenai/OLMo-core/pull/590
• Checkpointing improvements by @epwalsh in https://github.com/allenai/OLMo-core/pull/588
• GatedDeltaNet improvements by @tyler-romero in https://github.com/allenai/OLMo-core/pull/592
• Bump dataclass-extensions to fix pickle issue by @epwalsh in https://github.com/allenai/OLMo-core/pull/595
• Fix config encoding/merging with non-init fields by @epwalsh in https://github.com/allenai/OLMo-core/pull/596
• Internal cleanup of composable sampling classes by @epwalsh in https://github.com/allenai/OLMo-core/pull/599
• Fix config decode when CLASS resolves a Registrable subclass by @baileykuehl in https://github.com/allenai/OLMo-core/pull/600
• port over @dirkgr's fix for async callbacks by @epwalsh in https://github.com/allenai/OLMo-core/pull/601
• Fix GDN initialization by @tyler-romero in https://github.com/allenai/OLMo-core/pull/602
• Mark ephemeral checkpoints in metadata by @epwalsh in https://github.com/allenai/OLMo-core/pull/605
• Add claude md for Olmo-core by @tyler-romero in https://github.com/allenai/OLMo-core/pull/611
• Support flattening 3d parameters to 2d for use with Muon by @tyler-romero in https://github.com/allenai/OLMo-core/pull/606
• docs: SFT conversion tips for tokenizer override and parallel jobs by @natolambert in https://github.com/allenai/OLMo-core/pull/608
• updates to claude.md file by @epwalsh in https://github.com/allenai/OLMo-core/pull/612
• Support block-pattern based initialization of hybrid transformers by @tyler-romero in https://github.com/allenai/OLMo-core/pull/607
• Ladder5 v2 by @dirkgr in https://github.com/allenai/OLMo-core/pull/610
• Add model merging callback with integration test by @baileykuehl in https://github.com/allenai/OLMo-core/pull/558
• Fix A100 peak flops spec in speed monitor by @tyler-romero in https://github.com/allenai/OLMo-core/pull/614
• Add token ID validation guard in DataCollator by @tyler-romero in https://github.com/allenai/OLMo-core/pull/619
• fix typo in mixes init by @baileykuehl in https://github.com/allenai/OLMo-core/pull/621
• Support perplexity evals with context and tensor parallelism by @tyler-romero in https://github.com/allenai/OLMo-core/pull/616
• Add training smoketest skill for claude by @tyler-romero in https://github.com/allenai/OLMo-core/pull/618
• Hybrid model pre/mid/lc configs by @tyler-romero in https://github.com/allenai/OLMo-core/pull/613
• Add midtraining ingredient to to hybrid checkpoint list by @tyler-romero in https://github.com/allenai/OLMo-core/pull/633
• Add new in-loop eval tasks and update eval dependency to PyPI release by @kyleclo in https://github.com/allenai/OLMo-core/pull/587
• Full support for flash attention 4 (including generation kv cache support, installation from pypi) by @tyler-romero in https://github.com/allenai/OLMo-core/pull/642
• Add documentation for native generation module and chat interface by @tyler-romero in https://github.com/allenai/OLMo-core/pull/641
• Add Code Fresh per-language perplexity eval by @kyleclo in https://github.com/allenai/OLMo-core/pull/639
• fix attn backend forcing for TransformerGenerationModule by @tyler-romero in https://github.com/allenai/OLMo-core/pull/645
• ✨ Quality: Add documentation about checkpoint conversion limitations by @lukebaze in https://github.com/allenai/OLMo-core/pull/648

New Contributors

• @agupta2304 made their first contribution in https://github.com/allenai/OLMo-core/pull/473
• @ada-ggf25 made their first contribution in https://github.com/allenai/OLMo-core/pull/487
• @finbarrtimbers made their first contribution in https://github.com/allenai/OLMo-core/pull/497
• @jacob-morrison made their first contribution in https://github.com/allenai/OLMo-core/pull/400
• @natolambert made their first contribution in https://github.com/allenai/OLMo-core/pull/581
• @kyleclo made their first contribution in https://github.com/allenai/OLMo-core/pull/587
• @lukebaze made their first contribution in https://github.com/allenai/OLMo-core/pull/648

Full Changelog: https://github.com/allenai/OLMo-core/compare/v2.4.0...v2.5.0