Download Latest Version 0.5.4 source code.zip (4.8 MB) Google Add to Preferred Sources
Home / 0.5.4
Name Modified Size InfoDownloads / Week
Parent folder
0.5.4 source code.tar.gz 2026-08-26 2.3 MB
0
0.5.4 source code.zip 2026-08-26 4.8 MB
1 1 weekly downloads
README.md 2026-08-26 4.5 kB
0
Totals: 3 Items   7.2 MB 1

OpenCompass v0.5.4 Release Notes

🌟 Highlights

✨ Multimodal Evaluation with VLMEvalKit: OpenCompass now integrates VLMEvalKit for native multimodal dataset loading, OpenAI-compatible inference, and official VLMEvalKit evaluation metrics (#2586).

✨ Multi-Round Evaluation: GenInferencer now supports multi-round inference, accompanied by the new Multi-IF benchmark for evaluating multi-turn instruction following (#2557).

✨ Expanded Model API Ecosystem: Added LiteLLM as an AI gateway backend and extended TurboMind API capabilities with PPL evaluation support (#2441, [#2510]).

✨ Broader Benchmark Coverage: Added new evaluations spanning finance, long-context understanding, alignment, information extraction, and code reasoning, including BuySideFinBench, ELBench, Inverse IFEval, AA-LCR, CRUXEval, and updated LongBench/MRCR V2 configurations (#2446, [#2495], [#2514], [#2558]).

🚀 New Features

🔧 Integrated VLMEvalKit datasets and evaluators into the OpenCompass workflow, with MMBench and MMMU-Pro evaluation examples (#2586).

🔧 Added multi-round inference to GenInferencer and introduced Multi-IF dataset support (#2557).

🔧 Added LiteLLM AI Gateway model support and the TurboMind API /get_ppl endpoint (#2441, [#2510]).

🔧 Added and expanded benchmark configurations for CRUXEval, LongBench v1, Qasper, MRCR v2, Fake-Alignment, and more (#2572, [#2592], [#2549], [#2553]).

🐛 Bug Fixes

🔧 Strengthened evaluation correctness by rejecting deceptive return values in code evaluators and preventing invalid SIQA generations from being scored as correct (#2565, [#2450]).

🔧 Fixed generation behavior across HuggingFace and vLLM, including max_out_len=None, default torch dtype loading, accelerator generation config conversion, and chat-template BOS handling (#2459, [#1969], [#2519], [#2554]).

🔧 Improved evaluator reliability for BBH free-form answers, LiveCodeBench memory limits, Wikitext PPL without references, and official Fake-Alignment prompts (#2548, [#2538], [#2532], [#2567]).

🔧 Fixed dataset loading, local path resolution, download mappings, and prompts across AGIEval, BoolQ, GSM8K, C-Eval, SVAMP, TACO, and other benchmark configurations (#2497, [#2498], [#2540], [#2546]).

⚙ Enhancements and Refactors

⚙ Improved multi-round observability by fixing per-turn statistics in dump-res-length and repeat detection for multi-round conversations (#2569).

⚙ Improved model API robustness for Qwen response extraction, DeepSeek retry timeouts, Claude custom headers, and TurboMind proxy compatibility (#2521, [#2530], [#2445], [#2529]).

⚙ Improved data and task handling with safer SizePartitioner cost calculation and corrected TopK retriever metadata collation (#2438, [#2556]).

⚙ Refactored CI workflows, pinned third-party CI actions, and removed misleading error logs from successful RJob submissions (#2570, [#2533], [#2588]).

📦 Compatibility and Packaging

📦 Made OpenCC optional to restore Windows installation compatibility (#2512).

📦 Pinned the mmengine-lite runtime dependency for more reproducible installations (#2531).

📦 Resolved the Dingo/Pillow dependency conflict and fixed Dingo evaluation behavior (#2547, [#2607]).

🎉 Welcome New Contributors

A warm welcome and special thanks to our newest contributors who made this release possible:

  • @Amber0614 made their first contribution in (#2495).
  • @ziyangwang007 made their first contribution in (#2459).
  • @cindy90 made their first contribution in (#2446).
  • @irexyc made their first contribution in (#2510).
  • @Hibbert133 made their first contribution in (#2426).
  • @RheagalFire made their first contribution in (#2441).
  • @octo-patch made their first contribution in (#2438).
  • @HXC-Cloud-lab made their first contribution in (#2549).
  • @kobihikri made their first contribution in (#2533).
  • @thecaptain789 made their first contribution in (#2397).
  • @SongYippee made their first contribution in (#2445).
  • @hongleng made their first contribution in (#2556).
  • @thinszx made their first contribution in (#2586).
  • @Danielxu0208 made their first contribution in (#2578).

Full Changelog: https://github.com/open-compass/opencompass/compare/0.5.3...0.5.4

Thank you to all contributors who made OpenCompass v0.5.4 possible!

Source: README.md, updated 2026-08-26