Olmo-core Files

What's new

Fixed ✅

• Fixed parsing username+password git remote URLs in launch.beaker module.
• Fixed bug with default setup steps in launch.beaker.BeakerLaunchConfig when a branch can't be resolved.
• Cluster names in Beaker have changed.
• Fixed mixture rounding error with SourceMixtureDataset, which was previously causing samples to be repeated at the end of training.
• Don't DDOS Beaker from big jobs.
• A configuration error is now raised if you pass in a URL for the trainer or dataset's working directory. Previously the URL would just get mangled into a local path, leading to unexpected behavior.
• Fixed an issue where the ConsoleLoggerCallback would attempt to log before the first step.
• Only call teardown_distributed_environment() when training ends cleanly to avoid a hang for the duration of the distributed backend's timeout when there's an error from one rank.
• Fixed tensor parallelism issue with torch 2.8.
• More fixes for Beaker cluster names.
• Callback.post_train() will still be called even if the run is canceled before the dry-run batch.
• GarbageCollectorCallback will restore gc settings even when Trainer.fit() exits on an error.
• Make move_to_device blocking for MPS device to fix possible incorrect transfer of data from CPU to MPS.
• Fixed bug where glob_directory() would fail to match certain glob patterns.
• Added one more type of error to retry on when the Google Storage API throws it.
• Perform a garbage collection after checkpointing to avoid running out of CPU memory.
• Avoidable overflow error when using NumpyPackedFSLDataset.
• Fixed issue with NumpyFSLDatasetMixture + SourceMixtureDataset where not all instances would have the same sequence length.
• Attention backend will no longer default to flash in non-CUDA environments.

Changed ⚠️

• The dir option to Trainer.maybe_load_checkpoint() is now optional and defaults to the save_folder.
• Set fused_linear_cross_entropy_loss accum_dtype to fp32 in LMHead.
• Increased NCCL_FASTRAK_PLUGIN_ACCEPT_TIMEOUT_MS from 10 minutes to 30 minutes.
• SlackNotifierCallback will now notify on checkpoint saved and post epoch events.
• BeakerLaunchConfig.launch() will now send Slack notifications by default when follow=True if the env var SLACK_WEBHOOK_URL is set.
• src/examples/llama/ has been renamed to src/examples/llm/.
• Refactored eval task groups into task_groups.py
• The use_flash argument to the Attention classes is deprecated. Use backend="flash_2" instead.
• Refactored NumpyDatasetConfig by splitting it into a separate config per underlying dataset class.
• Refactored internal/experiment module to facilitate modifying datasets or supplying a fully custom ExperimentConfig.
• Simplified SourceMixtureDatasetConfig by removing redundant sequence_length and dtype fields.
• The model_id argument to convert_state_from_hf is deprecated. Conversion information is deduced from the model type.
• Refactored the example conversion scripts to/from HF, including decreasing false failures in validation.
• Small refactor to source_mixture.py to make it easier to define data mixes in yaml.
• Reorganized/cleaned up internal training scripts.

Added 🎉

• Added CLI script src/scripts/unshard.py for converting distributed checkpoints to regular PyTorch or safetensors format.
• Added a custom block that does LayerNorm scaling.
• Added OLMo-mix-0625-150Bsample data mix.
• Added alias support to DataMix enum.
• Added the HalfCos learning rate scheduler.
• Added CONTRIBUTING.md guidelines.
• Added a lightweight, gantry-like Beaker launch CLI: python -m olmo_core.launch.beaker.
• Added Beaker images with torch 2.8. There is olmo-core-tch280cu128-2025-09-18 and olmo-core-tch280cu129-2025-09-18 for CUDA 12.8 and 12.9, respectively.
• Added TransformerEngine to Docker images and a TransformerEngine attention backend.
• Added Callback.close() method, which is always called when exiting Trainer.fit().
• Added flash-attention 3 to Docker images, added flash_3 attention backend.
• Added support for sliding window attention to the Torch attention backend. Performance is not optimized, so other backends should be preferred.
• Added RoPEScalingConfig.to_hf_config() for each RoPE scaling method to support automatic conversion to HuggingFace format.
• Guide to dataset mixing in docs/source/guides/data_mixing.rst.
• Added support for converting FlexOlmo models (with both dropless and default MoEs) between OLMo Core and HF formats.
• Added olmo3_7B model config.
• Added additional internal configuration tools.
• Added a new named data mix that we used for the 32B run
• Added internal OLMo3 7B midtraining and long-context configs.
• Added ability to convert OLMo3 models to/from HF format with support for rope scaling configs.
• Added a script that can pull out a single training batch from a training job

Commits

5b324594 (chore) prepare for release v2.3.0 3f21b779 Reorganize internal scripts for Olmo3 (#423) 67dd1d64 Use exec to start torchrun (#422) 206d25ef Cookbook migration part 2 - long context config (#404) eabb8693 Raise timeout error if training job doesn't start in 5 mins (#421) cc69286c Script to dump training tokens (#418) b5ba7bef Script for unsharding (#420) 464d01e7 olmo3 conversion w/ support for rope scaling (#415) 3afa7ddb Add Rope scaling configs to rope module's exports (#414) 3ef0c050 Cookbook migration part 1 - midtraining config (#403) cdb79229 Add Dolma 3 sample, and a way to alias data mixes (#412) 77adc0bf pull Slack webhook URL from secret if available (#413) c8e32ab4 NumpyFSLDatasetMixture + SourceMixtureDataset fix (#411) 6ce62ccd Fix OverflowError in pack_documents (#409) e229c174 Cookbook migration part 0 - more setup (#397) 8eef6f36 Data mix for the 32B (#405) 9ba61540 Retry more on GS failures (#406) 73d11f8f Do GC after checkpointing (#407) cdfd2013 Typo f477a8d5 Fix glob_directory bug (#402) bc2a3f1c Support conversion of dropless MoE to FlexOlmo (#401) 7fc49de6 Support rope scaling configs for hf conversion (#394) 0f38dc6c HF Conversion Refactor (#390) a53f8259 source mixture dataset simplification and documentation (#399) 69bd9d2f MPS bug fixes (#395) 601d3367 refactor internal experiment configuration assembler (#386) 874be5d8 Add flash_3 attention backend (#377) 3aac6112 Remove old beaker refs (#393) 1aeb369c Add sliding window support for torch attention backend (#388) 7bebea39 Fully migrate to new cluster names, fix internal experiment launching on Augusta (#391) 1d4b67f8 Add flash-attn-3 install to dockerfile (#392) 04d5d5d6 Add Callback.close() method, other minor callback improvements (#389) dcf3a0eb Add attention backend abstraction and integrate transformer engine's attention (#384) 0356c1b1 Np Dataset Config Refactor (#381) afb58274 Improve distributed error handling (#380) 45d20313 Consolidate CLI code in public scripts (#379) cbcc7358 Port task groups from cookbook to OlmoCore (#378) 71f00237 bump torch and other dependencies in our Docker build (#360) b3c3b4e0 Add an all-in-one guide for researchers (#374) 65d411ba CONTRIBUTING.md (#376) e4695610 Treat cordoned beaker hosts as occupied (#375) 75d4d3a0 Avoid DDOS-ing Beaker for real (#372) 258a7a82 fix b068044d Set accum_grad to fp32 for fused_linear_cross_entropy_loss, bump liger-kernel version (#370) b550f9bf Add the ability to send Slack notifications from launch.beaker (#371) 801de4be Makes it possible to override the common config builder (#369) 59a2b2f0 Adds a new, highly specific LR scheduler (#368) d4ad23f7 fix rounding error with mixing datasets (#316) 46f7211d [Feat] Add LNS training example (#320) 3d9e9cdb redo base dir calculation for new cluster names (#364) b852ec04 Up NCCL_FASTRAK_PLUGIN_ACCEPT_TIMEOUT_MS to 30 minutes (#365) 72b7c1b9 Moved google-cloud-compute dependency from dev to beaker group. (#363) c00d715c catch issues with dolma metadata files earlier (#361) fa5a5dca Refine hostname constraints for beaker experiments on Google clusters (#355) 386b0a86 Fix parsing username+password git remote URLs (#356) 6726ffc4 make release process more robust