2.20.11 (2026-09-07)

Documentation

• docs: annotate **kwargs with Any in adding_a_dataset.md (#5395)

• docs: annotate **kwargs with Any in adding_a_dataset.md examples

• ci: skip tests for docs-only PRs

• Revert "ci: skip tests for docs-only PRs"

This reverts commit abe747bb3ed90d5bfffeee30b23cb1af9218d913. (ed47a25)

Fix

• fix: make model_meta and _set_seed torch-free (#5350) (ee03657)

Unknown

• Add model: Singaraj/morisien-embed-v1.5 (#5403)

Fine-tuned from LaBSE on the same Mauritian Creole corpus as Singaraj/morisien-embed, which was fine-tuned from multilingual-e5-base.

The first model is now marked superseded_by this one. On a retrieval pool built from FLORES+ with the released xSIM++ augmentation it scores below untrained LaBSE, while this one is the first configuration measured that beats it: error rate 0.2932 against 0.3343, McNemar exact p = 0.00083. (48c1f79)

• Add model: sshalimov04/ru-reranker-edge-150m (#5401)

• Add ModelMeta for sshalimov04/ru-reranker-edge-150m

• Add ModelMeta for sshalimov04/ru-reranker-edge-150m

• Add ModelMeta for sshalimov04/ru-reranker-edge-150m

• Add ModelMeta for sshalimov04/ru-reranker-edge-150m

• fix lint

---

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (79a9dad)

• dataset: add FineGrainOCR image-text clustering (#5341)

• Explore FineGrainOCR cross-modal clustering

• Build and benchmark FineGrainOCR clustering subset

• Move FineGrainOCR build scripts off task branch

• dataset: add FineGrainOCR image-text clustering

• dataset: use full FineGrainOCR validation split

• dataset: simplify FineGrainOCR task evaluation (5050aab)

• dataset: add Spoken Wikipedia speech-text retrieval (a2t, t2a, 5 languages) (#5368)

dataset: add Spoken Wikipedia speech-text retrieval (5 languages)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (5681a90)

• dataset: add Multi30k multilingual image-text retrieval (t2i, i2t) (#5322)

• dataset: add Multi30k multilingual image-text retrieval (t2i, i2t)

Adds Multi30kT2IRetrieval and Multi30kI2TRetrieval covering English, Czech, German and French. Neither direction was previously in mteb.

Multi30k stores one row per Flickr30k image with parallel human translations in four columns, so the image side is identical across language subsets. The loader builds it once per split and varies only the caption side.

Part of the multilingual coverage tracked in [#4842].

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

• fix: correct Multi30k bibtex order and exempt shared images

Two CI failures:

• BibTeX entries must be sorted by citation key, so barrault2018findings precedes elliott2016multi30k.
• The four language subsets intentionally share one 1000-image set, which trips the duplicate-image check. Exempted via KNOWN_ISSUES the same way XM3600T2IRetrieval and XFlickr30kCoT2IRetrieval already are.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

• refactor: use self.dataset with RetrievalSplitData for Multi30k

Addresses review feedback: the loader now writes task.dataset[lang][split] = RetrievalSplitData(...) instead of returning separate corpus/queries/relevant_docs dicts, and load_data on both classes delegates to the shared helper.

Also drops the modality columns and the None filler columns on both sides. They only existed to satisfy the old format; with RetrievalSplitData the modality follows from the columns present, so the image side is id + image and the text side is id + text.

Behaviour is unchanged - clip-vit-base-patch32 on en gives 0.7433 (t2i) and 0.7598 (i2t), identical to the scores in the PR description.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

• dataset: publish reshaped Multi30k and add construction script

Addresses review feedback asking for the source and a transformation script.

The task previously reshaped a third-party upload at load time. The reshaped form is now published in mteb's standard retrieval format, so the task file is metadata only and uses the default loader, and the transformation is reproducible from scripts/data/multi30k_retrieval/create_data.py rather than implicit in load_data.

Source is romrawinjp/multi30k at revision 110e827, MIT, unchanged. The image side is identical across the four language subsets, so it is stored once and every language config points at the same file rather than duplicating it.

The image table is written through the datasets API so the parquet carries the Image feature in its schema metadata; writing the struct directly produces a plain {bytes, path} column that does not decode on load.

Behaviour is unchanged: clip-vit-base-patch32 on en gives 0.7433 (t2i) and 0.7598 (i2t), identical to before, and the descriptive statistics are byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

• fix: correct license to cc-by-sa-4.0 and add W17-4718 citation for French extension

• fix: reformat BibTeX citations to pass citation formatting test

• fix: sort BibTeX fields alphabetically to pass citation formatting test

---

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (20c5a7b)

• model: add ImageBind model (Track 4, MOEB) (#5015)

• feat: add ImageBind model wrapper (Track 4, MOEB)

Adds ImageBindWrapper and ModelMeta for nielsr/imagebind-huge — Meta's joint multimodal embedding model supporting text, image, and audio in a single 1024-dim embedding space.

• Lazy import of imagebind package (optional dep group)
• Audio encoded via temp WAV files (same pattern as ebind_models.py)
• Images encoded via inline torchvision CLIP transforms (no temp files)
• Multi-modality fusion: element-wise sum + L2 normalize
• AudioCollator(target_sampling_rate=16_000) set when audio present
• Adds imagebind optional dep group to pyproject.toml
• Closes [#5014]

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

• fix: ruff lint fixes in imagebind_models.py

• N812: rename T alias to transforms (lowercase module as uppercase)

• PLW2901: suppress loop-var-overwrite warning with noqa on intentional reassign
• PTH108: replace os.unlink() with Path.unlink()
• E501: split long Resize() call across lines

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

• fix: ruff ICN001 - use from torchvision import transforms

• fix: sort imports in _load_images (isort I001)

• fix: remove temp WAV files, add source comment for image transforms

• fix: ruff format - wrap long get_clip_timepoints call

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

• get imagebind running on cluster

---

Co-authored-by: Rakshitha Ireddi <rakshithaireddi@Rakshithas-MacBook-Pro.local> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (61129c3)

• enable ANN for mteb/model_implementations (#5392) (03e73a4)