ggml Files

Overview

Version 0.23.0 adds new operators and backend scheduling APIs for sparse attention, asynchronous execution, and allocation-dependency tracking. It includes broad correctness and performance work across CPU, GPU, and accelerator backends, plus an expanded RPC stack with Apple RDMA support.

API changes

• Added GGML_GLU_OP_SWIGLU_CLAMP and ggml_swiglu_clamp() (llama/27930).
• Added ggml_flash_attn_ext_set_n_kv_max() for sparse K/V masks (llama/27970).
• RPC protocol bumped to 6.0.0 for event/async APIs (llama/18626).

Core changes

• Added SWIGLU_CLAMP and sparse-attention n_kv_max support (llama/27930, llama/27970).
• Added allocation-size expansion tracking and allocation-dependency support in backend graph optimization (llama/27960, llama/27301).
• Fixed multi-batch conv_transpose_2d and allowed SET_ROWS in backward passes (llama/26132).
• Backend search-path enumeration tolerates filesystem errors; meta buffers propagate usage (llama/28271, llama/27586).
• Replaced compile-definition versioning with generated ggml-version.h (llama/28364).
• RPC gains event/async APIs, Apple RDMA transport, and avoids serializing remote buffers (llama/18626, llama/26421, llama/26500).

Backend changes

CPU

• AVX2 IQ prompt speedups for large batches (llama/27402).
• Fixed s390x Q5_1 uninitialized accumulator and added conditional SpacemiT IME kernels (llama/28332, llama/27961).
• Reworked KleidiAI build/integration and avoided buffer-type init on dispatch (llama/26077, llama/27891).

CUDA / ROCm

• Added sparse FA, XOR-swizzled FA K/V smem, radix top-k, and extended MoE fusions (llama/27970, llama/25635, llama/27466, llama/25952).
• Added concurrent streams per split, MMQ/MMVQ tunings, and MoE MMQ fixes (llama/28198, llama/28285, llama/26264).
• ROCm/hip: tuned RDNA3 MMQ configs and optimized Q2_0 dot products (llama/26284, llama/26753).

Metal

• Added sparse FA, radix top-k, quantized concat, and chunked SSD MMA (llama/28098, llama/28073, llama/28116, llama/26647).
• Added fa-vec tunings across many Apple GPUs and Metal 4.0 tensor API support (llama/28373, llama/27461).
• Added metallib xcframework support and fixed memory, leak, and dispatch issues (llama/28163, llama/27758, llama/28306).

Vulkan

• Added cross-entropy loss, radix top-k, Lightning Indexer op, and IQ3_S batch mat-vec improvements (llama/27216, llama/28032, llama/27453, llama/27449).
• Fixed FA dequant, mul_mat_id padding, view-alias deps, and warp-size handling (llama/28190, llama/27925, llama/27812, llama/27726).
• Added shader hoisting for row IDs/expert counts and bfloat16 extension gating (llama/26686, llama/28155).

SYCL

• Added residual fusions, peer-to-peer copy, and host-pinned memory limits (llama/27610, llama/27550, llama/27559).
• Added GPU free-memory query, split TOP_K long rows, TILE quantized KV decode, and oneDNN SDPA f16 KV binding (llama/27968, llama/27847, llama/26689, llama/27468).
• Reduced redundant Q4_K MMVQ work and improved --fit handling (llama/27062, llama/27629).

OpenCL

• Added quantized lm_head/decode GEMV and medium-batch GEMM optimizations (llama/26477).
• Added Adreno fixes and matmul path improvements, Intel Xe-LP quant tuning, and MoE binary kernels (llama/27640, llama/26438, llama/27768).

Hexagon

• Added F16 unary ops, ABS/LOG, MUL_MAT/MUL_MAT_ID fusion, and CPY fence fixes (llama/28228, llama/27786, llama/28202, llama/28033).
• Added multi-NPU device support, fully async backend, and on-demand device discovery (llama/26501, llama/27785).

OpenVINO

• Updated OpenVINO, added new op support, whisper.cpp support, and a fuse-to-conv pass (llama/27843).

RPC

• Implemented event and async backend APIs and Apple RDMA transport (llama/18626, llama/26421).
• Avoided serializing remote buffers and fixed pre-RDMA macOS teardown/version issues (llama/26500, llama/27815, llama/27908).

WebGPU

• Fixed misaligned offset crash in tensor get and marked MUL_MAT for allocation expansion (llama/28045, llama/28071).

More info

• Releases and versioning of ggml-org projects

Changelog since v0.22.0

e91ded11 ggml : bump version to 0.23.0 (#1618) d8e66875 sync : llama.cpp 7523a092 metal : add remaining fa-vec tunings for M3 Max (llama/28373) 72d22bd4 ggml : replace compile definitions with version.h.in (llama/28364) 7aced598 ggml : rename and make private ggml_op_alloc_size_may_expand() (#0) 24b5deea sync : llama.cpp b849c94a ggml : don't crash when backend search path can't be read (llama/28271) da8baaba ggml : remove GGML_CUDA_PEER_MAX_BATCH_SIZE (llama/28177) 3573afc1 ggml-cpu(s390x) : fix q5_1 uninitialized v_acc (llama/28332) b70ff924 sycl: fuse rms_norm+mul+add and add+add residual chains (llama/27610) db62b97d SYCL: Refactor GGML_SYCL_ENABLE_MKL_FA to global var (llama/26863) 6921674b opencl: quant lm_head / decode GEMV and medium-batch GEMM optimizations (speculative decoding/MTP) (llama/26477) 90c4733f tune MMVQ to MMQ crossover for SM87 (llama/28285) 1145b17a metal : add sparse FA (llama/28098) 8ed43cf5 metal : fix glu dispatch with ne00 = 1 (llama/28306) eb624444 CUDA: Allow concurrent streams per split for multi-GPU (llama/28198) 355bd377 vulkan: fix FA dequant path engagement (llama/28190) 04c25ade sycl : enhance the api to support peer-to-peer copy (llama/27550) db157a7d sycl: reduce redundant work in Q4_K multi-column MMVQ (llama/27062) ee9842cb finetune: fix no KV cache (llama/27199) 8d6211cd ggml-hexagon: add F16 support for unary ops (llama/28228) 1e7de5d4 metal : add fa-vec tunings for M3 (llama/28236) f78f88b6 metal : fix memory query under low-memory conditions (llama/27701) b40e3c4d ggml-cuda : remove unused vars (llama/28235) 589cf31d CUDA + ggml: add sparse-fa for DSV4/GLM (llama/27970) b9ba0ae5 ggml: avoid KleidiAI buffer type init on dispatch (llama/27891) d9d16fda hexagon: MUL_MAT and MUL_MAT_ID fusion and fixes (llama/28202) a4325bb7 vulkan: handle larger batch sizes (>4) efficiently for IQ3_S mat-vec (llama/27449) bc194695 vulkan : only request VK_KHR_shader_bfloat16 extension if supported (llama/28155) ac6b4c67 ggml-cpu : conditionally add SpacemiT IME kernel sources (llama/27961) 902e018c opencl: fix out‐of‐bound reads in the Adreno image kernels (#27632) b201750b hexagon: add missing FARF logs for cpy/get_rows/set_rows/gdn ops (llama/28217) 4a53a5b8 metal : add metallib build support for xcframework (llama/28163) ee18d240 cuda: fuse MoE weighted expert reduction (llama/25952) 4af3ab1e Revert "sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 12…" (#28184) bed3f69a sycl : add Kronecker product FWHT support for sizes 384, 640, 768, 1280 (llama/28016) 4ca9b701 metal : add fa-vec tuning for M2 Pro (llama/28122) d6aeb74c metal : add fa-vec tunings for A18 Pro (MacBook Neo) (llama/28152) 0aa6c578 metal : fix more leaks due to missing autoreleasepools (llama/27883) 095419bf metal : add fa-vec tuning for M2 Max (llama/28015) 7641be24 sycl : support limit max alloc memory within 2GB for host-pinned memory (llama/27559) 4ac4c295 metal: enable Metal 4.0 tensor API on M5+/A19+ (llama/27461) 27438ee5 metal : add fa-vec tunings for M1 Ultra (llama/28088) fda7330d CUDA: XOR swizzle flash attn K,V smem fp16 tiles (llama/25635) f0251ebf metal : add concat support for quantized types (llama/28116) d2df1bc1 AVX2: Speed up large batch size prompt processing of IQ models (llama/27402) 5eee39f1 metal : add top-k radix implementation (llama/28073) 4ed94fa9 opencl: tune the quant paths for Intel Xe-LP GPUs to improve its TG and PP performance (llama/26438) 54787754 webgpu : avoid crash when offset is not multiple of 4 in WebGPU ggml_backend_tensor_get() implementation (llama/28045) ef26aaed ROCm: add radix TOP_K for long rows (llama/27466) 209bc903 metal : add fa-vec tunings for M1 (llama/28078) 45904618 CUDA: extend MOE fusion to specdec, earlier MOE glu fusion and topk-router fusion were restricted to 1 token (llama/27621) 0a97fdf4 sycl : Enhance to get the free memory of Intel GPU (llama/27968) b953b35f vulkan: tune mat-vec rows for batched inference on Strix Halo (llama/27909) 5bdb63f9 ggml : add MUL_MAT to the list of ops that may need additional memory (for WebGPU) (llama/28071) 8521c3be vulkan: top_k radix select for k >= 1024 for Qwen 3.8 Flash Next (llama/28032) f90ed9f8 hexagon: fix CPY fence bug (llama/28033) 2682b56f metal : add remaining Q4_1/Q5_0/Q5_1 fa-vec tunings for M2 (llama/28017) d4716378 sync : llama.cpp 9da56ef4 rpc: avoid serializing buffers from other servers (llama/26500) 6b44b36a ggml : fix ggml_backend_buft_get_alloc_size() guard (llama/28038) 7c06f018 ggml: add SWIGLU_CLAMP (llama/27930) f29745e7 CUDA: use the fast mm_ids_helper path for any n_expert_used (llama/27978) 985f2445 hip: tune rdna 3 mmq config (llama/26284) 6b9f44db hip : optimize Q2_0 dot-product path for gfx1201 (llama/26753) 7989099b sync : llama.cpp 57ce217e ggml : add ggml_backend_op_alloc_size_may_expand, use it in RPC (llama/27960) 43c95315 rpc: fix apple rdma error spew on teardown (llama/27908) d6fa5108 metal: add fa-vec tunings for M3 Ultra (llama/27999) b2fc6c42 metal : Add fa-vec tuning for M3 Pro (llama/27963) bb5b5ebb rpc : fix pre-rdma macOS versions (llama/27815) 3a2711cd hexagon: support for device discovery and create sessions on demand (llama/27785) db62fb37 sycl: split long rows in TOP_K instead of one work-group per row (llama/27847) 5033e538 metal : fix null-pipeline crash for F16 src1 mul_mat/mul_mat_id (llama/25648) 5600df56 ggml: allow passing alloc dependencies in graph_optimize (llama/27301) f494535b metal : add fa-vec tunings for M2 (llama/27940) 8fd7c641 opencl: use a better matmul path on two Adreno GPU generations (llama/27640) 7e3e8adf metal : assert shared memory padding (llama/27951) 309f6c7f metal : add remaining fa-vec tunings for M4 Pro (llama/27915) 1a82d669 sycl: make --fit respect --fit-target better (llama/27629) 503e6413 vulkan: combine duplicated fastdiv functions, rename the one optimizing small divs (llama/27526) 3c1172d0 metal : add fa-vec tunings for M1 Max (llama/27932) 8e6d7db7 vulkan: Change mul_mat_id to pad K rather than N (llama/27925) 8ec11f0a vulkan: fix missing view-alias dependencies in ggml_vk_graph_optimize (llama/27812) a573f763 ggml : fix conv_transpose_2d for multiple batches (llama/26132) 6555125d Vulkan: add hoisting support for row IDs and expert count in shaders (llama/26686) cfb61236 metal : add fa-vec tunings for M4 (llama/27875) bf88bab2 OpenVINO: Update OV to 2026.3.1, whisper.cpp support, Qwen3.5 on NPU, and new ops (llama/27843) 11a951be sycl: use TILE for quantized KV decode on BMG (llama/26689) 31cd2ca7 sycl: bind the f16 KV cache in place for the oneDNN SDPA path (llama/27468) f1354dce metal : add fa-vec tunings for M3 Max, M5 and M5 Pro (llama/27863) 15bfbd28 metal : add fa-vec tunings for M4 Pro (llama/27824) 28e9bae3 ggml-hexagon: add HTP unary ops for ABS and LOG (llama/27786) 96e52581 hex-unary: fix RMS_NORM_MUL weight-offset bugs for grouped/broadcast norms (llama/27798) 202d0779 spec : add DFlash2 support (local convolution + candidate selector) (#27342) (llama/27816) 9f321690 opencl: add bin kernels kernel_gemm_moe_q4_0_q8_1_dp4a_bin, kernel_gemm_moe_mxfp4_q8_1_dp4a_bin (llama/27768) 161c8296 Feature: Added LIGHTNING_INDEXER support for Deepseek V4 ops on Vulkan Backend (llama/27453) 09eaacf5 metal : fix memory leaks due to missing autoreleasepools (llama/27758) fee87ab0 hexagon: support for multi-NPU devices (IQ9, IQ10) and fully asynchronous backend (llama/26501) acb2e0d9 vulkan: warptiles currently assume warp sizes <= 64, clamp to work around larger warps (llama/27726) f49b86e5 Implemented vulkan cross_entropy_loss and cross_entropy_loss_back (llama/27216) 82d091a7 rpc : implement event and async backend APIs (llama/18626) 41a2a154 sync : llama.cpp 098d3552 cuda: unblock mmq for MoE on sm_60 (llama/26264) 68fca9ef sync : llama.cpp ecaf424e ggml-metal: add chunked SSD MMA for Mamba-2 prefill optimization (llama/26647) d845b717 ggml-meta: propagate buffer usage and call init on the new tensors (llama/27586) 359a1df1 kleidiai: Rework KleidiAI Build System/Integration (llama/26077) 96557922 rpc: support apple RDMA as an RPC transport (llama/26421) e9c3b152 metal : null-check buffer alloc to fix OOM crash (llama/25371) 36da5713 scripts : add a prompt for generting release summary