2.20.13 (2026-09-14)

Fix

• fix: declare ModelMeta load dtypes with OutputDType instead of torch.dtype (#5441)

• fix: declare ModelMeta load dtypes with OutputDType instead of torch.dtype

• address copilot comments

• address remaining copilot review comments

• clarify OutputDType docstring: not all members map to a same-named torch dtype (INT4/UINT4/BINARY are compression-only levels)

• note in compress_embeddings.md that the ValueError for an invalid output_dtype is raised at encode() time, not construction
• fix AST guard in test_ensure_no_torch_at_import.py to track function body nodes by identity instead of line number, so a one-line function signature default is no longer mistaken for its body

• broaden the guard's dtype whitelist to every torch.<name> attribute that is a torch.dtype, catching aliases like torch.half/torch.long

• lint and format

---

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (c4f968c)

Unknown

• dataset: add OmniWikiRetrieval v2i and v2t from Tevatron Wiki-SS-NQ (#5415)

dataset: add OmniWikiRetrieval (1443790)

• move Nano retrieval tasks to the v2 dataset format (#5452)

refactor: move Nano retrieval tasks to the v2 dataset format

All 13 Nano* tasks carried an identical v1 load_data() override that read the zeta-alpha-ai copies into self.corpus/self.queries/self.relevant_docs. The re-uploads under the mteb org are in the v2 layout, so the override can go and the default AbsTaskRetrieval loader handles them.

Verified per task that the v2 dataset loads identical content to the original: corpus ids and text, query ids and text, and qrels all match for 13/13. The v1 override hardcoded every relevance score to 1; the uploaded qrels also carry score 1 throughout, so scores are unchanged.

Refs [#3424]

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (f18a069)

• Add tencent/EVIE-8B and tencent/EVIE-4.5B (#5450)

• Add tencent/EVIE-8B and tencent/EVIE-4.5B

Co-authored-by: Cursor <cursoragent@cursor.com>

• Fix ruff literal-membership in EvieWrapper

Co-authored-by: Cursor <cursoragent@cursor.com>

• Declare evie extra as conflicting and update uv.lock

evie pins transformers>=5.13.1, which is incompatible with visrag-ret and other extras (transformers<4.53). Add evie to [tool.uv] conflicts so uv can resolve, and regenerate the lockfile.

Co-authored-by: Cursor <cursoragent@cursor.com>

• Address review: optional FA2, simpler bidir call, drop flash-attn from evie extra

• FA2 now optional: warn and fall back to attn_implementation='sdpa' when flash_attention_2 is unavailable, instead of raising.

• Call self.model.enable_bidirectional_attention() directly; drop the getattr/callable guard and the manual _enable_bidirectional_attention helper.
• Remove flash-attn from the evie extra (FA2 is an optional perf add-on) and regenerate uv.lock.

Co-authored-by: Cursor <cursoragent@cursor.com>

• Delete mteb_mock_run_results.md

• fix typing

---

Co-authored-by: Cursor <cursoragent@cursor.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a25a026)

• dataset add ColDeRReranking benchmark (#5409)

• feat(tasks): add ColDeRReranking benchmark (#2709)

• test: add duplicate_text exemption for ColDeRReranking in task quality tests

• fix(reranking): clarify ColDeR benchmark semantics

• fix: normalize ColDeR citation formatting

• fix(reranking): remove unused ColDeR logger

• Apply suggestion from @Samoed

---

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (5427a1f)

• model: Add litillabs/litil-embed-0.6b (#5456)

• Add litillabs/litil-embed-0.6b

• Remove mock run report (40a77d2)

• model: Add models multi-modal-embed [MOEB] (#5011)

• [MOEB] Add models multi-modal-embed

• adding n_embedding parameter

• fix large model

• fix lint

• robustness auto model or model

• lock update + remove numpy cast + shared base for large and small + remove _COMMON

• remove librosa, cherry pick custom code from large model, and remove patches

• remove labels

• simplifying implementation and pinning revisions

• fix lint

• update number of parameters (ef8b40f)

• task: add COCO Modality Equivalence retrieval tasks (#5384)

• feat: add COCO Modality Equivalence retrieval tasks (issue [#5358])

• fix: bibtex formatting and ruff format in coco modality equivalence

• Sort bibtex entries alphabetically and add blank lines between them so bibtexparser round-trip matches the test expectation

• Collapse multi-line author fields to single line (bibtexparser strips continuation indentation, breaking the equality check)
• Reformat chained cast_column calls in create_data.py to satisfy ruff format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

• fix: correct BEIR format and per-config dataset routing

• Each task class now specifies its own dataset config name (t2i, a2i_h, a2i_s, i2t, i2a_h, i2a_s) via _dataset() helper; the shared _COMMON dict no longer contains the dataset key

• Rewrite create_data.py to load sources using the BEIR corpus/queries/qrels config structure; fix corpus split name for mbeir_mscoco_task0 ("corpus" not "test"); build all six BEIR-format DatasetDicts and save locally before push

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

• Fill dataset revision and fix build script for COCO modality equivalence
• Set _DATASET_PATH to rakshi719/coco-modality-equivalence
• Set _DATASET_REVISION to [3aabaa]
• Fix Image/Audio decode=False to avoid PIL/torchcodec encoding issues
• Fix ID normalization: extract 12-digit COCO IDs from image paths
• Fix MSCOCO loading: use 'query' config (not 'queries') for text captions
• Fix push format: 18 HF configs ({task}-corpus/queries/qrels) matching MTEB loader

• Add Audio.encode_example monkeypatch for datasets 5.0.0 torchcodec issue

• Fix ruff formatting in create_data.py

• Add COCO modality equivalance task, analysis script and gap results

• Add directional asymmetry analysis script and results (issue [#5360])

• Add sampling budget analysis script and results (issue [#5362])

• feat(analysis): PCA latent dimensions analysis for MTEB issue [#5367]

• Fix COCO modality-equivalence retrieval tasks and add descriptive stats

• Add load_data override so the HF config prefix (t2i/a2i_h/a2i_s/i2t/i2a_h/i2a_s) is actually used instead of always requesting the default config.

• Fix create_data.py: a2i_h/a2i_s qrels referenced prefixed query-ids while the queries split used unprefixed ids, leaving 0/120 retrievable queries; rebuilt and pushed corrected dataset to mteb/coco-modality-equivalence.
• Add generated descriptive stats for all six task directions.

• Reorder all entries in tasks/retrieval/eng/init.py (ruff --fix).

• Point analysis scripts at fixed dataset and rerun results

• Update _REPO in directional_asymmetry.py, modality_gap.py, and sampling_budget.py from rakshi719/coco-modality-equivalence to the corrected mteb/coco-modality-equivalence.

• Rerun all four analysis scripts (directional_asymmetry, modality_gap, sampling_budget, latent_dimensions) and refresh their results/*.json.

---

Co-authored-by: Rakshitha Ireddi <rakshithaireddi@Rakshithas-MacBook-Pro.local> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (1dfd69a)

• Create Ops MM models (2B, 7B) (#5332)

• Create ops_mm_models

Co-Authored-By: Deep Shah <21212684+deep9539@users.noreply.github.com>

• Update ops_mm_models.py

• lint

---

Co-authored-by: Deep Shah <21212684+deep9539@users.noreply.github.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (4e24e0c)

• Add WeMM embedding model. (#5333)

• Add WEMM models

• Update pyproject.toml

• Use InstructSentenceTransformerModel

• fix video and better support instruction prompt

• Update wemm_models.py

• Update wemm_models.py

• reformat and fix batch padding

• fix padding

• fix padding

• Update wemm to use sentence_transformer

• Add init method

• Reformat to pass linter

• Add comment to explain the permute

• address comments

---

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (951d86b)

• Add RzenEmbed model (#5339)

• first commit

• handle empty task instruction

• resolve naming discrepancy

• fix things

• add tensor support

• dimension handling

• fix error

• visual embeddings to language model dimension

• Update rzen_embed_model.py

Fix system prompt, image + video coprocessing.

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

• delete not required files.

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

• resolve comments

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

• fix error

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

• update image embed dimension

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

• Add change to make rzen embed compatible to >= 4.57.0

• add rzen training data and embedding_params

• lint

• Update rzen_embed_model.py

---

Co-authored-by: nehal309 <20415842+nehal309@users.noreply.github.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (74ee71c)

• dataset: add ToolRetrieval benchmark task (#5410)

• feat(retrieval): add ToolRetrieval benchmark task (#3628)

• fix(retrieval): reproduce published ToolRet results

ToolRetrieval scored ~11 NDCG@10 above the paper because it differed from the reference implementation in three ways, each making retrieval easier:

• queries retrieved from their own category's tools rather than the pooled 44,453-tool corpus, which is what eval_retrieval(category="all") does upstream and what the published tables report
• subset scores micro-averaged over queries instead of taking the unweighted mean over the 35 retrieval tasks; this dominates web, whose tasks range from 11 to 1,100 queries
• instructions joined as "{query} {instruction}" rather than the reference template "Instruct: {instruction}\nQuery: {query}"

Pool the corpus, expose the 35 retrieval tasks as subsets so MTEB's cross-subset mean matches the paper's aggregation, and add ToolRetrievalInstruction for the w/ inst. setting (Table 5) alongside ToolRetrieval (w/o inst., Table 4).

all-MiniLM-L6-v2 w/ inst., NDCG@10, paper Table 5 in parentheses: web 13.23 (12.77), code 32.38 (31.59), customized 33.29 (32.24). Mean absolute delta drops from 10.91 to 0.76. Reproduce with scripts/reproduce_toolret.py, which also raises max_seq_length to 512 as the reference implementation does; this model's card ships 256 and web swings ~5 NDCG@10 on that alone.

• feat(retrieval): reproduce ToolRet paper trends in the example script

The example script now carries the reference implementation's per-model handling, without which the published numbers do not come out: SentenceTransformer vs fp16 AutoModel dispatch, per-family pooling, the L2-normalization skip for contriever and gtr-t5, per-family prompt templates, min(max_position_embeddings, 2048), and word-level truncation. It takes --model/--all/--settings and prints a PASS/FAIL against the published values, defaulting to Table 5.

Over 9 baselines the paper's findings replicate:

• instructions help every model, mean gain +11.26 NDCG@10 vs the paper's +11.34 (Pearson 0.958 across models)
• the model ranking replicates (Pearson 0.988, Spearman 0.967)
• subset difficulty orders the same way, web < code < customized
• 85% of metrics land within 2.0 NDCG@10, mean absolute delta 1.39; 92% for w/ inst. excluding Tool-COLT, whose public checkpoint appears not to be the one the authors evaluated

Exact per-cell agreement is not available: the reference print_results() computes a size-weighted (micro) mean while the published tables match an unweighted (macro) one, so the paper's numbers were not produced by the released code.

• fix(scripts): correct batching and refuse unsafe model fallbacks

Two bugs in the reproduction script, both found by running it over every baseline rather than one:

• the token-budget batch size was computed from the shortest text in a sorted batch, so short queries produced enormous forward passes and tripped a device-side assert; it is now capped at MAX_BATCH, and the SentenceTransformer path sizes its batch from max_seq_length instead of a hardcoded 32, which was exhausting a 24GB card at 2048 tokens
• a trust_remote_code=False fallback was added for checkpoints whose remote code lags transformers, but for gte-Qwen2 that loads a different model: its custom code makes attention bidirectional where the native architecture is causal, and it scored ~19 NDCG@10 below the published value while looking plausible. The fallback is removed so the load fails loudly instead.

Coverage is now 10 of Table 5's 26 rows -- every single-vector dense retriever under 7B that runs in this harness. gtr-t5-large is included despite scoring below gtr-t5-base, inverting the paper's ordering for that pair; it is reported as measured rather than dropped, and the docstring records both the headline figures and the figures excluding it.

• fix(scripts): use native MTEB ToolRetrieval evaluation

• chore: remove Modal evaluation helper

• reupload

• remove unnecessary script

---

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a768963)

• Add ModelMeta for NGA-KR/ko-embed-cls (#5446)

• Add ModelMeta for NGA-KR/ko-embed-cls

• Fix ModelMeta for NGA-KR/ko-embed-cls (lint, model_type, prompts)

• Add n_embedding_parameters

---

Co-authored-by: Isaac Chung <isaac.chung@foam.io> (cb65aad)

• task: add BioVITA multimodal retrieval (#5153)

• task: add BioVITA multimodal retrieval

• fix: use fixed-format datasets for BioVITA

• fix: simplify BioVITA task metadata and loading

• fix: classify BioVITA tasks as reranking

• fix: address BioVITA reranking review

• style: format BioVITA data script

• fix: return BioVITA audio fallbacks

• fix: add BioVITA descriptive statistics

• test: update BioVITA reranking quality allowlist

• docs: clarify BioVITA tie handling

• refactor: remove _BioVITAReranking base class, add task_specific_scores to each class explicitly

Also fix BibTeX formatting in webvid_covr files to satisfy pre-commit hook.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---

Co-authored-by: Michelle Yang <myang333@users.noreply.github.com> Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (6ee088e)

• Add ModelMeta for NGA-KR/ko-embed-nli (#5447)

• Add ModelMeta for NGA-KR/ko-embed-nli

• Add n_embedding_parameters (8b85f40)

• add Nemotron model citations and public training references (#5445)

• fix: add Nemotron citations and Embed-VL training code link

• fix: link Nemotron 3 public training datasets

• fix: link the full Nemotron 3 training data sections (4fe4737)

• add OpenMDW license metadata for corresponding models (#5444)

fix: update OpenMDW license metadata for Nemotron models (5826242)

• recognize OpenMDW-1.1 as an open license (#5443)

fix: recognize OpenMDW-1.1 as an open license (1183517)

• Add ModelMeta for NGA-KR/ko-embed-v0 (#5440)

• Add ModelMeta for NGA-KR/ko-embed-v0

• lint

---

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a8bce9d)

• Update commit hash for FlowVec (#5439) (4b868d2)