Download Latest Version v1.1.0 source code.zip (3.5 MB)
Email in envelope

Get an email when there's a new version of WhisperKit

Home / v1.1.0
Name Modified Size InfoDownloads / Week
Parent folder
README.md < 10 hours ago 13.1 kB
0
v1.1.0 source code.tar.gz < 10 hours ago 3.3 MB
0
v1.1.0 source code.zip < 10 hours ago 3.5 MB
0
Totals: 3 Items   6.9 MB 0

Highlights

  • WhisperKit: Significant peak memory reduction via incremental audio file loading: 70%+ savings for 3-hour audio input.
  • TTSKit: ~40% end-to-end speedup with improved inference for Qwen3-TTS.
  • SpeakerKit: speaker centroid embeddings are now available from a public interface, useful for matching speakers across separate diarization runs.
  • Bug fixes for promptTokens, Chinese word timestamps, transcribeWithOptions, TTS chunk boundaries, and the model load/cache path.

[!WARNING] TTSKit's default model variants changed (a one-time download on upgrade) and speech generation now requires macOS 15 / iOS 18. The previous assets remain supported - see Model Assets.

WhisperKit: Incremental File Loading

By default, WhisperKit decodes an entire audio file into memory before transcribing, which can lead to OOM situations for multi-hour audio files. .incremental streams it from disk in bounded-memory chunks instead:

:::swift
let pipe = try await WhisperKit()

let results = try await pipe.transcribe(
    audioPath: "path/to/large-audio.wav",
    audioInputOptions: AudioInputOptions(audioLoadingMode: .incremental)
)

Chunks are cut at silence (VAD) boundaries, so the transcript matches a full-file run with chunkingStrategy: .vad - only peak memory differs. Tune with .incremental(chunkDuration:chunkBufferSize:) (defaults: 120s chunks, 2 chunks buffered at a time).

Also available via CLI:

:::bash
swift run argmax-cli transcribe \
  --model large-v3-v20240930_626MB \
  --audio-path "path/to/large-audio.wav" \
  --incremental-loading

This will become a default option in a future release, so let us know how it works for you!

TTSKit: Faster Models

Both Qwen3-TTS decoders now ship as multifunction Core ML assets, which lets us load slightly different model forward pass configurations using the same set of weights.

SpeechDecoder (TTSKitConfig.speechDecoderMode): .latencyOptimized (default, 1 audio frame/call, ~80ms in length) for lowest time-to-first-audio, or .throughputOptimized (4 frames/call, ~320ms) to speed up throughput overall at the cost of a slower initial buffer.

MultiCodeDecoder (TTSKitConfig.multiCodeDecoderMode): a talker frame expands into 15 residual codes. .stepped (default) does that in 16 Core ML calls with host-side sampling; .fused does the whole frame in one call with sampling and lookups inside the graph.

Benchmarks in #494 and #513.

:::swift
// Defaults: latency-optimized SpeechDecoder, stepped MultiCodeDecoder
let tts = try await TTSKit()

// Opt in
let config = TTSKitConfig(
    speechDecoderMode: .throughputOptimized,
    multiCodeDecoderMode: .fused
)
let fasterTTS = try await TTSKit(config)

Via CLI:

:::bash
swift run -c release argmax-cli tts --text "Hello there." \
  --speech-decoder-mode throughputOptimized \
  --multi-code-decoder-mode fused --play

The mode is read once at model load since it's considered an entirely separate model to the OS. Set it before constructing TTSKit, or reload to switch at runtime. The two modes don't produce byte-identical audio, but it's audibly indistinguishable - fused sampling draws a different but equally valid sequence. The TTSKitExample app gains sidebar pickers for both modes.

The layout is detected from the asset at load time, so the legacy single-function W8A16 variants keep working (#520) - see Model Assets.

SpeakerKit: Speaker Centroid Embeddings

DiarizationResult now carries speakerCentroidEmbeddings: [Int: [Float]] which can be used to link speakers across separate diarization runs without re-running the embedder:

Within a single result - compare two local speaker ids:

:::swift
let result = try await speakerKit.diarize(audioArray: multiSpeakerAudioFloats)
if let distance = result.centroidCosineDistance(between: 0, and: 1) {
    // Distance ranging from 0 - 2, where 0 is an exact match
    print("speakers 0 and 1: cosine distance \(distance)")
}

Across separate runs - each diarize(...) assigns its own local speakerIds (0, 1, …). To link speakers in a later chunk to an earlier one, take a centroid from the earlier result and pass it to nearestSpeakerCentroid(to:) on the later result:

:::swift
let speakerKit = try await SpeakerKit()
let meeting1 = try await speakerKit.diarize(audioArray: meeting1Floats)
let meeting2 = try await speakerKit.diarize(audioArray: meeting2Floats)

// Centroid from an earlier run for speaker 0, can be nil
guard let speakerCentroidMeeting1 = meeting1.speakerCentroidEmbeddings[0] else { return } // handle no speakers found

// Which meeting2 speaker sounds most like meeting1 speaker 0?
if let match = meeting2.nearestSpeakerCentroid(to: speakerCentroidMeeting1) {
    print("meeting1 speaker 0 most similar to meeting2 speaker \(match.speakerId) (distance \(match.distance))")
}

Centroids are in raw embedder space, so pick your own distance threshold that works best for your data.

Model Assets

TTSKit's default variants moved to multifunction assets:

Component Before After
SpeechDecoder W8A16 W8A16-multifunction
MultiCodeDecoder W8A16 W8A16-multifunction

Both are published on argmaxinc/ttskit-coreml and download automatically on your next setupModels() / first launch from huggingface. Expect a one-time download on upgrade if the models were downloaded previously, or:

Keeping the previous W8A16 assets

The decoders detect the model types at load time, so the legacy single-function variants keep working if you pin them to the pre-v1.1.0 variants:

:::swift
let config = TTSKitConfig(
    speechDecoderVariant: "W8A16",
    multiCodeDecoderVariant: "W8A16"
)
let tts = try await TTSKit(config)

If the assets are already in your cache, this will use them instead of downloading the new models.

The legacy assets only implement the default modes. Requesting .throughputOptimized or .fused against them throws TTSError.invalidConfiguration naming the multifunction variant to use, rather than silently falling back.

New minimum OS for TTSKit

TTSKit speech generation now requires macOS 15, iOS 18, watchOS 11, or visionOS 2 — the decode path moved to MLTensor, and the pre-macOS 15 path was removed. This applies to both asset layouts, so pinning W8A16 doesn't avoid it; loadModels() throws TTSError.modelLoadingFailed on older OS versions. The package's declared platforms are unchanged (iOS 16 / macOS 13) and WhisperKit and SpeakerKit are unaffected - but a TTSKit app targeting iOS 17 will build and then fail at model load. We recommend available flags on your TTS entry points with #available(iOS 18, macOS 15, *), if you support iOS 17 or below.

API Changes

Deprecations

  • AudioInputConfig -> AudioInputOptions (typealias kept).
  • WhisperKit.audioInputConfig -> WhisperKit.audioInputOptions.

  • WhisperKitConfig.audioInputConfig -> pass audioInputOptions per call to transcribe(...). The stored value still works as the instance default.

    :::swift // before let config = WhisperKitConfig(audioInputConfig: AudioInputConfig(channelMode: .sumChannels(nil))) let pipe = try await WhisperKit(config) let results = try await pipe.transcribe(audioPath: path)

    // after let pipe = try await WhisperKit() let results = try await pipe.transcribe( audioPath: path, audioInputOptions: AudioInputOptions(channelMode: .sumChannels(nil)) )

Breaking changes

Callers of TTSKit are unaffected, but if you have a custom class that conforms to the SpeechDecoding protocol, the following changes are needed:

  • decodeFrame(codes:cache:) / decodeFrameAsync(codes:cache:) take codes: [[Int32]] instead of [Int32] - an outer array of codesPerStep frames.
  • New required codesPerStep: Int, read from the loaded model's audio_codes input shape.

New CLI flags

argmax-cli transcribe --incremental-loading
                      --incremental-chunk-duration <seconds>
                      --incremental-chunk-buffer-size <count>

argmax-cli tts --speech-decoder-mode      latencyOptimized|throughputOptimized
               --multi-code-decoder-mode  stepped|fused

Community

Big thanks to everyone who shipped code, filed issues, and dug into reproductions for this release. 🙏

  • The fused MultiCodeDecoder proposed by @mjfrey's #506 landed, collapsing the frame into one call with in-graph Gumbel-max sampling, which gave a nice speedup to the QwenTTS pipeline.
  • @leecrossley contributed speaker centroid embeddings (#463), closing a long-standing request for cross-run speaker matching, with unit and integration tests.
  • @freecodetiger tracked down the NLLanguage normalization bug that had been breaking Chinese word timestamps for quite some time (#511).
  • Thanks to @hakanensari, @yangzichao, @sborisov88, and @alan890104 for reproductions and investigation on the promptTokens empty-transcription bug (#514).

This is a big release so let us know how it goes in your testing. Open an issue or join us in Discord. 🚀

What's Changed

New Contributors

Full Changelog: https://github.com/argmaxinc/argmax-oss-swift/compare/v1.0.0...v1.1.0

Source: README.md, updated 2026-08-06