LiteRT Files

LiteRT Release Notes: Version 2.3.0

Summary

• Faster LLM inference : FlashAttention/Flash-Decode and fused GPU kernels bring up to ~38% faster decode and ~3× faster long-context prefill.
• Lower memory/model size: Saves >4 GB RAM for 32K attention; Intel weight sharing cuts Gemma 4 12B AOT size by ~45%.
• Better GPU/NPU support: Improved Metal memory residency, zero-copy cache, broader MediaTek NPU support, and cleaner vendor packaging.
• Platform/API updates: New Swift APIs, separate Android GPU package, LiteRT Support migration, and Android minSdk 24.
• Testing: Broader ATS operator and macOS GPU coverage.

1. LiteRT Runtime

⚡ Performance and Memory

• Fused FlashAttention-2 Prefill, Flash-Decode & GQA Support (odml.sdpa_transposed)
  • Implemented single-kernel Flash-Decode and FlashAttention-2 Prefill GPU kernels on Metal.
  • These optimizations increase Qwen3-0.6B decode speed by ~38%, accelerate 32K long-context prefill by 2.98x on Qwen3-0.6B and 1.40x on Qwen3-4B. (Tested on MacBook M4 pro) and save >4 GB of peak RAM at 32K context by avoiding full attention score matrix materialization.
• Fused LLM & Hybrid Architecture Composite Kernels (odml.qkv_norm_rope, odml.swiglu):
  • QKV-RMSNorm-RoPE (odml.qkv_norm_rope) & SwiGLU (odml.swiglu): Added GPU composite kernels (Metal and OpenCL) and opt-in GraphFloat32/IrModel runtime graph fusion passes that fuse QKV splitting, per-head Q/K/V RMSNorm, and RoPE into a single kernel and fuse gate-up MLP activations, reducing Qwen3 decode operator dispatches by 69% and improving decode throughput by ~12%.
• GPU Memory Residency
  • Dynamic Metal Memory Residency (MTLResidencySet): Added runtime GPU memory residency support on macOS and iOS to dynamically lock and unlock model weights and mutable external KV-cache tensors in physical RAM across inference sessions, preventing OS swap paging on memory-constrained devices.

🛠️ Hardware Acceleration

• Intel NPU/GPU: Added support for weight-sharing. Cross-signature weight sharing roughly halves the size of AOT-compiled LLMs on Intel NPU: Gemma4 12B's prefill_decode model drops from ~11.6 GB to 6.3 GB (~45%).
• GPU: Use zero-copy mmap for SerializationProgramCache. This reduced peak RAM footprint for some large model’s inference.

🔌 APIs & Platform support

• GPU: Release symbolized GPU Accelerators for easier debugging.
• GPU: Android GPU Maven package is released separately as “litert-gpu”.
• Skip applying NPU accelerator in CompiledModel when model has no NPU ops
• Swift APIs: A type-safe Swift wrapper over the core C API, enabling iOS, macOS, and cross-platform apps to load, compile, and execute LiteRT models with hardware acceleration (developer guide).
• Support Library: migrate from TensorFlow Lite Support to LiteRT Support (migration guide).
• Android: minSdk is bumped from 23 to 24.
• Android: MediaTek NPU support is expanded to all Android S+ versions.
• Android: vendor specific NPU Maven packages are provided. Litert_npu_runtime_libraries.zip is no longer supported. To build NPU libs as Google Play Feature Delivery, see https://github.com/google-ai-edge/litert-samples/tree/main/samples/litert/image_segmentation/kotlin_npu for the reference.

🐛 Security, Vulnerability and Stability

• ATS: Defined nomenclature CoreSingleOp for 20 key ops; expanded op coverage to all core single ops.
• Expanded ATS platform coverage to MacOS (Metal & WebGPU).