The interactive file manager requires Javascript. Please enable it or use sftp or scp.
You may still browse the files here.

Name	Modified	Size	InfoDownloads / Week
Parent folder
2.20.12 source code.tar.gz	2026-09-09	14.4 MB	0
2.20.12 source code.zip	2026-09-09	16.9 MB	0
README.md	2026-09-09	7.6 kB	0
Totals: 3 Items		31.3 MB	0

2.20.12 (2026-09-09)

Ci

ci: Pin bibtexparser below 2.0.0 (#5423)
Initial plan
Pin bibtexparser below 2.0

Co-authored-by: isaac-chung <48971969+isaac-chung@users.noreply.github.com>

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: isaac-chung <48971969+isaac-chung@users.noreply.github.com> (692bc29)

Fix

fix: make abstasks and model wrappers torch-free (#5436) (6618bf5)

Unknown

make _evaluators and _create_dataloaders torch-free (#5426)
fix: make _evaluators and _create_dataloaders torch-free
revert mteb import
revert whitespace changes
lintter (3f90c53)
Include all benchmark memberships in task scores (#5425)

fix: include hidden benchmarks in task->benchmark membership index

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (df7998e)

[MOEB] task: add EPICKitchens100Clustering (#5221)
[MOEB] task: add EPICKitchens100Clustering (video clustering, closes [#5219])

25 verb-class categories, 500 clips (20 per class), sampled from the lightly-ai/epic-kitchens-100-clips mirror of EPIC-KITCHENS-100 (CC-BY-NC-4.0, not gated). Cluster labels use each verb_class's canonical (most frequent) surface form, since individual narrations use varied verb strings for the same class (e.g. both "pour-out" and "pour-from" fall under one verb_class) -- the class id, not any single row's raw verb field, is the real cluster label.

Verified end-to-end: real data loads correctly (500 rows, 25 classes, 20/class). Two independent real evaluations confirm a consistent, non-degenerate signal:

microsoft/xclip-base-patch16: v_measure=0.3367 (std 0.0081)
microsoft/xclip-base-patch32: v_measure=0.3189 (std 0.0098)
docs: add EPICKitchens100Clustering construction script

Addresses Samoed's review request for the data transformation script.

feat: add --full mode for complete 25-class EPIC-Kitchens clustering dataset

Use a single glob pattern for snapshot_download in --full mode instead of one exact-filename pattern per clip -- with 34,564 patterns checked against 37,455 remote files, huggingface_hub's pattern matching becomes O(patterns x files) and effectively hangs. Since --full already wants 92% of the clips/ folder, downloading it wholesale is both simpler and avoids the blowup.

docs: add full-vs-sample v_measure comparison results
task: split EPICKitchens100Clustering into full and mini variants

isaac_chung's review found the previously-shipped -Full dataset upload was broken: only 1.19MB instead of ~20GB. Root cause was running the build script with system python3 (no torchcodec) instead of the project's .venv -- push_to_hub()'s external-file embedding for the Video() feature silently no-ops without a working video decoder, so every row fell back to a bare path reference instead of real embedded bytes. Rebuilt and repushed with the correct interpreter; verified 20.32GB across 32 files this time.

EPICKitchens100Clustering now points at the full 34,564-clip dataset; EPICKitchens100ClusteringMini is the previous 500-clip downsampled variant, renamed per review. Also removes epic_full_vs_sample_results.json per review (the PR description already reports those numbers).

test: fix kwargs type annotation, add descriptive stats for full task

Fixes the lint failure (missing-type-kwargs) that surfaced after rebasing onto main's newer lint rules. Also adds descriptive_stats for the full EPICKitchens100Clustering task now that the dataset is fixed.

style: fix ruff formatting in create_data.py (efa8063)
Update FlowVec commit hash (#5420)
Add LingoIITGN/qwen-indic-v1 to sentence_transformers_models.py
Update mteb/models/model_implementations/sentence_transformers_models.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

Update ModelMeta with from_hub hardware values
format
linting issues fix
Update commit hash for FlowVec

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (8808bbc)

Add FlowVec-v1 (#5417)
Add LingoIITGN/qwen-indic-v1 to sentence_transformers_models.py
Update mteb/models/model_implementations/sentence_transformers_models.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

Update ModelMeta with from_hub hardware values
format
linting issues fix
Add ModelMeta configuration for indicflow-v1
Add ModelMeta configuration for indicflow-v1
Update sentence_transformers_models.py
lint

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (bc0d4f8)

[MOEB] task: add LibriSpeechSpeakerA2ARetrieval (audio→audio retrieval open slot. Closes [#2311].) (#5224)
[MOEB] task: add FSD50KA2ARetrieval (audio-to-audio retrieval, closes [#5222])

100 queries / 200 corpus docs, 20 sound-event classes (5/10 per class), sampled from FSD50K's public eval split. Relevance = same primary-class membership; queries and corpus are disjoint clips (no duplicate audio).

Verified: random baseline ndcg@10=0.0586 (matches base rate), real evaluation with mispeech/dasheng-base ndcg@10=0.2343 (hit_rate@1=0.32 vs random's 0.05) -- clear non-degenerate signal.

docs: add FSD50KA2ARetrieval construction script

Addresses Samoed's review request for the data transformation script.

task: replace FSD50KA2ARetrieval with LibriSpeechSpeakerA2ARetrieval

Both reviewers pushed back on repurposing FSD50K's existing classification labels as retrieval relevance -- it tests the same signal as the FSD50K classification task already in mteb, adding little benchmark value.

Replaces it with a genuinely native audio-to-audio task: given a speech clip, retrieve other clips from the same speaker in LibriSpeech test-clean. This is a real, distinct task family (speaker re-identification), not adapted from any existing classification labels, and not redundant with LibriTTS's existing a2t/t2a transcription-retrieval tasks in mteb.

200 queries / 400 corpus docs / 2000 qrels, all 40 test-clean speakers (min 32 clips each), 5 queries + 10 corpus docs per speaker, CC-BY-4.0. Verified: random baseline ndcg@10 0.031 (near chance), mispeech/dasheng-base ndcg@10 0.899 -- clear, genuine signal.