OpenCompass v0.5.0 Release Notes

🌟 Highlights

✨ ​Comprehensive Scientific Benchmarks: Integrated 10+ specialized datasets (MedXpertQA, ClimaQA, SmolInstruct, etc.), covering multiple scientific fields such as chemistry, physics, biology, and earth sciences ✨ Cascade Evaluator: Supported cascading eval methods from rules to LLM judgments. ✨ New Runner: Supported for Rjob Runner has now been completed. ✨ OpenAISDK Streaming: Provided a more stable OpenAI API method. ✨ New Evaluation Examples: Published the real-time evaluation config of CompassAcademic Leaderboard and the Intern-S1 related benchmark evaluation config.

🚀 New Features

🔧 Cascade Evaluator (#1992) 🔧 Rjob Runner (#2144) 🔧 OpenAISDK Streaming (#2208) 🔧 Evaluation Example for CompassAcademic Leaderboard. (#2202) 🔧 Evaluation Example for Intern-S1 and Scientific Benchmarks. (#2220) 🔧 So Many New Scientific Datasets! 1. MedXpertQA for expert-level medical knowledge evaluation (#2002)
2. ClimaQA for climate question evaluation (#2017) 3. HealthBench for better measuring capabilities of AI systems for health (#2099) 4. ProteinLMBench for protein related tasks (#2064) ...

📖 Documentation

📝 Fixed 404 links between Chinese/English docs (#2001)
📝 Added CompassAcademic Leaderboard task tutorial (#2202) 📝 Added Intern-S1 evaluation task tutorial (#2220) 📝 Fixed format problems of the dataset statistics page (#2170) 📝 Align NIAH CLI command guide to the actual CLI argument parser (#2194) 📝 Set correct paths for the examples (#2198)

🐛Bug Fixes

🔧 Fixed compare error base_evaluator (#2010) 🔧 Fixed OpenICL Math Evaluator Config (#2007) 🔧 Added Error Case for content filter (#2167) 🔧 Fix the OpenAI SDK to adapt to gpt-5 (#2236) 🔧 Fixed dataset repeat by concatenating (#2039) 🔧 Concat OpenaiSDK reasoning content (#2041)

⚙ Enhancements and Refactors###

⚙ Infrastructure Refactors: - Set dump-eval-details as default behavior (#1999)
- Refactorized openicl eval task (#1990)
- Added openai_extra_kwargs for API customization (#2210)

⚙ CI/CD Improvements: - Fixed baseline score (#2000) - Updated baseline for kernal change of vllm and lmdeploy (#2011) - Updated baseline and fix lmdeploy version (#2098) - Added check rule (#2101) - Updated testcases' baseline (#2184) ...

🎉 Welcome New Contributors

A warm welcome to our newest contributors: - @Yejin0111 for MedXpertQA clinical dataset (#2002) - @smgjch for matbench development (#2021) - @taolinzhang for rewardbench dataset (#2029) - @xiexinch for fixing lawbench evaluation (#2037) - @xuxuxuxuxuxjh for ClinicBench, PubMedQA and ScienceQA datasets (#2061) - @mar-cry for NEJM AI benchmark (#2063) - @bio-mlhui for CARDBiomedBench dataset (#2071) - @tchenglv520 for Lifescience subset support for MMLU & SciEval (#2059) - @Flaick for MMLU Pro Biomedical version support (#2081) - @yuehua-s for o4-mini model (#2083) - @kkscilife for adding CI check rule (#2101) - @yusun-nlp for SmolInstruct dataset (#2127) - @soki123 for SRbench dataset (#2105) - @suencgo for PHYBench dataset (#2125) - @Zhouzone for updating Earth Silver benchmark (#2140) - @uyzhang for R-Bench dataset (ICML 2025) (#2091) - @f14-bertolotti for stabilizing MBPP evaluation (#2111) - @fly2tomato for debugging Rjob runner (#2171) - @debuggingworld for fixing Qwen3 model config field error (#2152) - @blueternalness for aligning NIAH CLI command guide (#2194) - @KADCA21 for BlueLM-2.5 API (#2193) - @dbinthesky for KCLE feature (#2224) - @FarongWen for EESE dataset and configs (#2223) - @mazihan880 for CodeCompass dataset and configs (#2214)

Full Changelog: https://github.com/open-compass/opencompass/compare/0.4.2...0.5.0

Thank you for using OpenCompass! These updates empower deeper insights and more reliable evaluations. Keep exploring, and stay tuned for future innovations! 🌟