2.18.12 (2026-08-02)

Documentation

• docs: add multiple choice retrieval task example (#4931)

Add a dedicated example to the 'Adding a Task' docs showing how to implement a multiple choice retrieval task. It explains that such tasks compare each query against a fixed candidate set via the top_ranked split and use accuracy as the main score, and demonstrates building the top_ranked split from relevant_docs in dataset_transform, following the BLINK task pattern.

Closes [#4408] (3748f63)

Fix

• fix: Correctly use .n_active_parameters so that active parameters correct when using the API (#5066)

Active parameters is incorrect when using the API Fixes [#5064]

@Samoed anything I am missing here? (ee5689e)

Unknown

• Fix/afrie5 loader kwargs + changing AfriE5 revision (#5059)

• fix: pass dtype via model_kwargs and drop invalid normalized kwarg for AfriE5-Large-instruct

• fix: bump AfriE5-Large-instruct revision to include the modules.json,sentence_xlm-roberta_config.json, and 1_Pooling/config.json file additions

---

Co-authored-by: Nicolas Helmeyer <helmeyen@login-3.server.mila.quebec> Co-authored-by: Nicolas Helmeyer <helmeyen@login-1.server.mila.quebec> (d0bad1e)

• Add Hanno-Labs/dinghy-law-8b-v1 (legal embedding model) (#5058) (4ce470f)

• Move data to mteb HF repo (mmvu, covers80, flare, miao, seavl, shs100k) (#5056) (0456b96)

• benchmark: Expand MTEB(kor, v*) with retrieval/STS/NLI/clustering tasks (#4870)

• Add 7 Korean retrieval tasks to MTEB(kor, v1)

Extend MTEB(kor, v1) retrieval coverage with LawIRKo, SQuADKorV1Retrieval, AutoRAGRetrieval, PublicHealthQA, BelebeleRetrieval, MultiLongDocRetrieval, and MrTidyRetrieval (kor subset), matching the KURE Korean retrieval set.

• Add Korean STS/NLI/clustering tasks to MTEB(kor, v1)

Extend MTEB(kor, v1) with STS17 (ko-ko), KLUE-NLI and PawsXPairClassification (NLI / pair classification), and SIB200ClusteringS2S, KlueMrcDomainClustering, KlueYnatMrcCategoryClustering (clustering). Kor-NLI is omitted (no mteb task).

• Add ModelMeta for 9 Korean community embedding models

Register dragonkue/BGE-m3-ko, dragonkue/multilingual-e5-small-ko, dragonkue/snowflake-arctic-embed-l-v2.0-ko, exp-models/dragonkue-KoEn-E5-Tiny, jhgan/ko-sroberta-multitask, nlpai-lab/KURE-v1, nlpai-lab/KoE5, telepix/PIXIE-Rune-v1.5, upskyy/bge-m3-korean so their Korean results render on the leaderboard. Metadata fetched from the HF Hub; loaders mirror each base family.

• Support Korean (kor) in bm25s baseline tokenization

Add 'kor' to the bm25 language map with character-level tokenization (matching the existing jpn/zho handling for no-space scripts), so the bm25s reference baseline can index Korean retrieval tasks in MTEB(kor, v1).

• Address review: MTEB(kor, v2), rename korean_models.py

• Create MTEB(kor, v2) with the expanded task list instead of mutating MTEB(kor, v1). v1 is reverted to its original 6 tasks (Korean Legacy, alias 'MTEB(kor, classic)'); v2 holds the 19 tasks and takes the 'MTEB(kor)' alias. Leaderboard menu points the language-specific entry at v2 and keeps v1 under the legacy 'Other' section, mirroring MTEB(eng, v1/v2).

• Rename korean_finetuned_models.py -> korean_models.py.

(n_embedding_parameters is already set on all 9 entries.)

• Actually add MTEB(kor, v2) + menu registration (was missing from prior commit)

The prior commit only captured the file rename; the benchmarks.py / init.py / _leaderboard_menu.py edits were not staged. This commit adds them:

• MTEB(kor, v2) with the expanded task list; MTEB(kor, v1) reverted to its original 6 tasks (Korean Legacy, alias 'MTEB(kor, classic)').
• Register MTEB_KOR_V2 in benchmarks init (import + all).

• Leaderboard menu: v2 in language-specific, v1 in legacy 'Other'.

• Fill training_datasets where documented (review follow-up)

Sources: each model's card at the pinned revision (+ KURE GitHub repo).

• upskyy/bge-m3-korean, jhgan/ko-sroberta-multitask: trained on KorSTS(+KorNLI; no mteb task) -> {"KorSTS"}.
• dragonkue/multilingual-e5-small-ko, exp-models/dragonkue-KoEn-E5-Tiny, dragonkue/snowflake-arctic-embed-l-v2.0-ko: AI Hub Korean MRC corpora only (none are mteb datasets) -> empty set, so base-model data is still inherited via adapted_from.
• dragonkue/BGE-m3-ko, nlpai-lab/KURE-v1, telepix/PIXIE-Rune-v1.5: fine-tuning data unnamed in the card -> left None (unknown).

• nlpai-lab/KoE5: trained on nlpai-lab/ko-triplet-v1.0 (linked via public_training_data) whose source composition is undocumented -> left None.

• BGE-m3-ko: training_datasets set() (author-confirmed, no mteb data in fine-tune)

• Move pixie_rune_v1_5 to pixie_models.py (review request)

Relocate the telepix/PIXIE-Rune-v1.5 ModelMeta from korean_models.py to pixie_models.py alongside PIXIE-Rune-v1.0, reusing that file's PIXIE_RUNE_V1_PROMPTS (identical query-prefix scheme). Entry values unchanged.

• Fix max_tokens: 8194 -> 8192 for all affected entries

8194 is max_position_embeddings (incl. XLM-R's 2 offset slots), not the usable input length: these models' tokenizer_config model_max_length and sentence-transformers max_seq_length are both 8192, which is what ModelMeta.max_tokens is documented to mean. Fixes the three Korean bge-m3 fine-tunes (per review suggestion) and the same error inherited in BAAI/bge-m3 itself, manu/bge-m3-custom-fr, GreenNode VN x2, AITeamVN/Vietnamese_Embedding, deepvk/USER-bge-m3, jina-embeddings-v3.

• Add KorNLI to MTEB(kor, v2)

KorNLI is now available in mteb; include it in the v2 pair-classification set (v2: 19 -> 20 tasks). Merged main to pick up the task definition.

• Move new model registrations out to [#4921] (review request)

Per review, the 9 Korean community ModelMetas now live in [#4921] so that embeddings-benchmark/results#581 can merge independently; this PR stays a benchmark-definition change (MTEB(kor, v2), bm25s Korean tokenization, max_tokens fixes).

• MTEB(kor): mark v1 superseded by v2, describe v1->v2 changes

Address review on [#4870]:

• MTEB(kor, v1): aliases=[], display_name="Korean", add superseded_by=["MTEB(kor, v2)"]
• MTEB(kor, v2): expand description to state what changed since v1 (grows 6 -> 20 tasks)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---

Co-authored-by: dgyu <dgyu@sionic.ai> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> (db49990)

• model: Add dragonkue/multilingual-e5-small-ko-v2 and sionic-ai/comsat-e… (#5054)

• Register dragonkue/multilingual-e5-small-ko-v2 and sionic-ai/comsat-embed-ko-8b-preview

Two Korean community embedding models evaluated on MTEB(kor, v2):

• dragonkue/multilingual-e5-small-ko-v2 (multilingual-e5-small finetune)
• sionic-ai/comsat-embed-ko-8b-preview (Qwen3-Embedding-8B finetune)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

• Address review: mark e5-small-ko superseded, trim comsat-ko comment

• dragonkue/multilingual-e5-small-ko: add superseded_by=multilingual-e5-small-ko-v2

• sionic-ai/comsat-embed-ko-8b-preview: remove redundant loader comment (per review)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> (53a9c40)

• Add perplexity-ai/pplx-embed-v1-late-0.6b model meta (#4813)

• Add perplexity-ai/pplx-embed-v1-late-0.6b model meta

Closes [#4691]. Adds the ModelMeta entry for perplexity-ai/pplx-embed-v1-late-0.6b, a PyLate/ColBERT late-interaction (MaxSim) embedding model with 128-dim token-level vectors, continued-trained from perplexity-ai/pplx-embed-v1-0.6b. Metadata sourced from the HF model card/API and config files (revision, MIT license, 128-dim projection, 512 document length, ~596M params).

• Add embedding parameter count for pplx late embed (7c2712d)

• dataset: Repoint dead dataset to mteb's supported for AfriSentiLangClassification (#4912)

• Fixed AfriSentiLangClassification: repointed dead HausaNLP dataset to mteb/afri_senti_lang mirror, removed redundant tweet->text preprocessing step

• remove processing

---

Co-authored-by: Nicolas Helmeyer <helmeyen@login-2.server.mila.quebec> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (74044b9)