Summary

Overview

Requirement changes. hydra-core and omegaconf are now core dependencies (they were in espnet[train]). espnet[asr] now takes ctc-segmentation from upstream 1.7.5, replacing the espnet-ctc-segmentation fork. New extras: espnet[splet] and espnet[rst].

What is new.

• Audio quality assessment: espnet2.bin.aqa_inference, the ARECHO metric model, and the URGENT24 recipe (#6751-#6759). This is what VERSA needs to install its supporting models from PyPI rather than from a git clone.
• Bagpiper on the command line (espnet describe, espnet render) and in the MCP server.
• Scaled dot-product attention through every ASR encoder, relative-position attention included, and in the TTS, SLU, speaker and UniVERSA encoders.
• F5-TTS models, and the Phonological Tokenizer (parts 1-4).
• ESPnet3: inference, dataset, metric and stage contracts; configurable global batch distribution; loading a published bundle by its Hugging Face Hub tag.
• A long list of decoding, streaming and training fixes; see Bugfix below.

Important PRs

New Features

• PR [#6515]: [espnet3-recipe][f5-tts 2] Add F5-TTS models and Helpers (by @NewGamezzz)

Recipe

• PR [#6947]: Bump the version to 202611 (by @sw005320)
• PR [#6931]: [espnet3] Add Contracts across infrence, publication, and demos (by @Masao-Someki)
• PR [#6917]: [espnet3] Restore publication/demo stage (by @Masao-Someki)
• PR [#6913]: Say what the ESPnet vLLM fork serves: Bagpiper-TTS for the demo, and OWSM (by @sw005320)
• PR [#6911]: Remove ESPnet3 inference declarations and retain independent workflows (by @sw005320)
• PR [#6895]: [espnet3] Add configurable global batch distribution to ESPnet3 (by @Masao-Someki)
• PR [#6893]: [espnet3] Add contract for stages (by @Masao-Someki)
• PR [#6888]: Move the SpeechLM shell helpers under scripts/ (by @sw005320)
• PR [#6885]: [espnet3] Write a Demos notebook from the system's Inference declaration (by @sw005320)
• PR [#6880]: [espnet3] Write a model card's usage from the system's Inference declaration (by @sw005320)
• PR [#6853]: [espnet3] Add a dataset contract (by @Masao-Someki)
• PR [#6851]: Fix Bagpiper dependency compatibility and dataset CLI routes (by @jctian98)
• PR [#6843]: [espnet3] Add contracts on Metrics (by @Masao-Someki)
• PR [#6841]: [espnet3] Rename the tts system to f5tts (by @NewGamezzz)
• PR [#6839]: [espnet3] Test collect_stats, infer and measure on two CPU workers (by @sw005320)
• PR [#6827]: Rename audio assessment components to AQA with legacy aliases (by @ftshijt)
• PR [#6822]: Fix Bagpiper public-checkpoint training, resume, and inference (by @jctian98)
• PR [#6811]: Say what POWSM-CTC is, and what each control does (by @sw005320)
• PR [#6810]: Bagpiper on the command line and the MCP server (by @sw005320)
• PR [#6806]: [espnet3] Derive the infer stage from the Inference declaration, and make load() the one way to read a bundle (by @sw005320)
• PR [#6795]: [espnet3] Rename asr system to esp2_asr (by @Masao-Someki)
• PR [#6759]: [Audio metrics 12/12] Finish AQA entry-point cleanup and verify checkpoints (by @ftshijt)
• PR [#6758]: [AQA 11/12] Add URGENT24 audio quality assessment training examples (by @ftshijt)
• PR [#6757]: [Audio metrics 10/12] Add a tiny AQA integration recipe (by @ftshijt)
• PR [#6756]: [Audio metrics 09/12] Add shared audio metric recipe scaffolding (by @ftshijt)
• PR [#6755]: [AQA 08/12] Add the staged audio quality assessment recipe driver (by @ftshijt)
• PR [#6754]: [Audio metrics 07/12] Add metric vocabulary and evaluation utilities (by @ftshijt)
• PR [#6655]: Add seoul corpus ASR recipe (by @KimHaeri)
• PR [#6636]: Sidon Recipe Continued (by @ZhuoyanTao)

Bugfix

• PR [#6921]: Accept numeric custom logarithm bases in mel features (by @antonsoo)
• PR [#6919]: Keep UtteranceMVN gradients finite for constant channels (by @antonsoo)
• PR [#6870]: Ignore blank lines of the non-linguistic symbols file in the tokenizers (by @antonsoo)
• PR [#6869]: Return frame lengths from HuggingFaceFrontend for waveform models (by @antonsoo)
• PR [#6865]: Do not train another epoch when resuming a run that was stopped early (by @antonsoo)
• PR [#6863]: Do not leave extra spaces when char tokenization removes non-linguistic symbols (by @antonsoo)
• PR [#6860]: Keep dots in the utterance id when writing inference artifacts (by @antonsoo)
• PR [#6855]: Average n-best models from the checkpoints as they were saved (by @antonsoo)
• PR [#6854]: [espnet3] Collect statistics without the train preprocessor's augmentation (by @sw005320)
• PR [#6852]: [espnet3] Bugfix: collect_stats shapes were omitting text shapes (by @chenehk)
• PR [#6848]: [bugfix] espnet3 checkpoints unable to resume from their own checkpoints when using PiecewiseLinearWarmupLR (by @chenehk)
• PR [#6824]: Make FoldedBatchSampler honour drop_last and min_batch_size (by @MohammadHijjawi97)
• PR [#6823]: Check language codes against the checkpoint before decoding (by @meryemsakin)
• PR [#6819]: Mask every frame of a zero-length sequence in make_pad_mask (by @MohammadHijjawi97)
• PR [#6817]: Take the audio feature extractor without the video one (by @sw005320)
• PR [#6816]: Match a longer tokenizer symbol before a prefix of it (by @SashaMIT)
• PR [#6813]: fix(speechlm): make synchronize_batches() equalize the batch count across ranks (by @li-lizhe)
• PR [#6803]: Retry a 401 only where no credentials were sent (by @sw005320)

Documentation

• PR [#6941]: Bump vite to 8.3.5 in doc/vuepress (by @sw005320)
• PR [#6938]: Bump markdown-it to 14.3.3 and 15.0.3 in doc/vuepress (by @sw005320)
• PR [#6894]: espnet3: lift Field, kinds and check_declaration to espnet3.api; call api "interfaces" (by @sw005320)
• PR [#6881]: [cli] Add espnet run: any published model, by what it declares (by @sw005320)
• PR [#6874]: Raise the PR size guideline to 30 files / 1500 lines (by @sw005320)
• PR [#6847]: Bump source-map-js to 1.2.2 in doc/vuepress (by @sw005320)
• PR [#6840]: Move hydra-core and omegaconf into the core dependencies (by @sw005320)
• PR [#6814]: Fix docs SSR retention during the VuePress upgrade (by @meryemsakin)
• PR [#6805]: [espnet3] Add the inference contract (espnet3.api.inference) and the ASR system's implementation (by @sw005320)
• PR [#6753]: [Audio metrics 06/12] Add audio metric inference and model packing (by @ftshijt)
• PR [#6752]: [Audio metrics 05/12] Register the audio_metric training task (by @ftshijt)
• PR [#6751]: [Audio metrics 04/12] Add ARECHO autoregressive metric model (by @ftshijt)

Others

• PR [#6946]: Pass use_sdpa through the TransformerEncoder for multi-speaker ASR (by @li-lizhe)
• PR [#6936]: Pass use_sdpa through the speaker MfaConformerEncoder (by @emirmrkaljevic)
• PR [#6933]: Pass use_sdpa through the SLU ConformerPostEncoder (by @emirmrkaljevic)
• PR [#6930]: Remove non-working automatic Codex review workflow (by @sw005320)
• PR [#6928]: Avoid NaN gradients in AAM-softmax losses at |cos| == 1 (by @sw005320)
• PR [#6927]: Fix Codex trigger comment permissions (by @sw005320)
• PR [#6926]: Fix automatic Codex review trigger (by @sw005320)
• PR [#6924]: Add automatic Codex review request workflow (by @sw005320)
• PR [#6923]: Depend on upstream ctc-segmentation 1.7.5 (by @lumaku)
• PR [#6922]: Let LabelAggregate frame labels with center=False (by @MohammadHijjawi97)
• PR [#6918]: Preserve GlobalMVN statistics across mixed precision calls (by @antonsoo)
• PR [#6916]: docs: add temporary Codex review guidance via CONTRIBUTING.md (by @sw005320)
• PR [#6909]: Fix RNN/TCN separator and ConvDecoder streaming paths (by @Cassian433)
• PR [#6906]: Pass use_sdpa through the SLU TransformerPostEncoder (by @emirmrkaljevic)
• PR [#6902]: README: move ESPnet2/ESPnet3 config differences below Contributing, collapsed (by @sw005320)
• PR [#6901]: [espnet3] Load a pack_model bundle by its Hugging Face Hub tag (by @Shunsuke32)
• PR [#6900]: [espnet2] Make the number and dilations of ECAPA-TDNN blocks configurable (by @Shunsuke32)
• PR [#6891]: Pass use_sdpa through ARUniversa (by @MannXo)
• PR [#6890]: Pass use_sdpa through UniversaBase (by @mangeshraut712)
• PR [#6889]: Add use_sdpa to the GST style encoder (by @emirmrkaljevic)
• PR [#6886]: Let relative-position attention use SDPA, and wire use_sdpa through every ASR encoder (by @sw005320)
• PR [#6884]: Preserve audio tensor views when framing streaming STFT (by @antonsoo)
• PR [#6883]: Preserve device and dtype in streaming STFT buffers (by @antonsoo)
• PR [#6882]: Handle omitted lengths when merging streaming STFT chunks (by @antonsoo)
• PR [#6879]: [mcp] Run the MCP speech-to-text tools through the espnet3 API (by @sw005320)
• PR [#6878]: [cli] Run espnet transcribe, translate and phonemize through the espnet3 API (by @sw005320)
• PR [#6877]: [espnet3] load() on device="auto": CUDA when torch sees it, else the CPU (by @sw005320)
• PR [#6876]: Let a config reach MultiHeadedAttention's use_sdpa (by @sw005320)
• PR [#6873]: [espnet3] Add the OWSM system's Inference (esp2_s2t), stream esp2_asr's blockwise decoder (by @sw005320)
• PR [#6872]: Point the TTS demo badge at the Space this repository maintains (by @sw005320)
• PR [#6871]: Report the mean of the non-target scores as nontrg_mean in the speaker validation stats (by @antonsoo)
• PR [#6868]: Stop end detection from waiting out the primer (by @antonsoo)
• PR [#6867]: Keep the correct causal context in Conv1d streaming (by @antonsoo)
• PR [#6866]: Fix the output lengths of the asr_transducer encoder with Conv1d blocks (by @antonsoo)
• PR [#6864]: Score a prefix longer than the encoder output as impossible (by @antonsoo)
• PR [#6862]: Fix the output mask of the asr_transducer ConvInput block (by @antonsoo)
• PR [#6861]: Keep the language and task positions out of the long-form CTC posteriors (by @antonsoo)
• PR [#6859]: Give FusedFrontends the length of each utterance (by @antonsoo)
• PR [#6858]: Do not overwrite sample_rate when enh_scoring resamples for PESQ (by @antonsoo)
• PR [#6857]: Keep the hyp primer out of the CTC prefix score (by @antonsoo)
• PR [#6856]: Let the CTC window cross a silence longer than itself (by @antonsoo)
• PR [#6850]: [espnet3] Let a system's Inference read the models its backend publishes (by @sw005320)
• PR [#6849]: Add inplace true for relu functions (by @sabaebrahimi)
• PR [#6837]: Make the fileio readers' in check the scp keys (by @MohammadHijjawi97)
• PR [#6836]: Give rst_inference's parser a description, which the doc build asserts (by @sw005320)
• PR [#6832]: Add SPLET, the text-side counterpart of VERSA, as a skeleton (by @sw005320)
• PR [#6831]: Catch a token layout that moved under a re-saved config (by @sw005320)
• PR [#6829]: Constrain hydra targets when loading an untrusted publication bundle (by @sw005320)
• PR [#6825]: Annotate what load_rttm_text returns, so RttmReader can load a file (by @MohammadHijjawi97)
• PR [#6820]: Fix the format string VADScpWriter writes each segment with (by @MohammadHijjawi97)
• PR [#6815]: Prevent Hugging Face decoder overrides from enabling custom code (by @meryemsakin)
• PR [#6808]: fix(speechlm): make synchronize_batches work on non-CUDA accelerators (by @li-lizhe)
• PR [#6804]: Say each thing once, and stop hard-wrapping the prose (by @sw005320)
• PR [#6802]: Report after the workflows it reports on, not before them (by @sw005320)
• PR [#6801]: Retry a 401 from the Hub, which is not about credentials (by @sw005320)
• PR [#6799]: The last two cells are green (by @sw005320)
• PR [#6797]: [espnet3] fix hollow tests and close coverage gaps from unit-test review (by @Masao-Someki)
• PR [#6731]: Select an available quantized engine before quantize_dynamic (by @amirlankalm)
• PR [#6730]: Let batch_decode take a padded numpy array or a list of utterances (by @amirlankalm)
• PR [#6729]: Vectorise ScoreFilter.batch_score (by @amirlankalm)
• PR [#6699]: [Phonological Tokenizer 4/7] Add task and trainer for Phonological Tokenizer. (by @ondatk68)
• PR [#6698]: [Phonological Tokenizer 3/7] Add espnet_model for Phonological Tokenizer (by @ondatk68)
• PR [#6696]: [Phonological Tokenizer 2/7] Add objectives for fine-tuning the tokenizer (by @ondatk68)
• PR [#6637]: [Phonological Tokenizer 1/7] Add differentiable k-means and tokenizer class (by @ondatk68)

Contributors

22 contributors, across 124 merged pull requests in v.202611.

@amirlankalm, @antonsoo, @Cassian433, @chenehk, @emirmrkaljevic, @ftshijt, @jctian98, @KimHaeri, @li-lizhe, @lumaku, @mangeshraut712, @MannXo, @Masao-Someki, @meryemsakin, @MohammadHijjawi97, @NewGamezzz, @ondatk68, @sabaebrahimi, @SashaMIT, @Shunsuke32, @sw005320, @ZhuoyanTao.

Full changelog: https://github.com/espnet/espnet/compare/v.202610.post2...v.202611