ggml Files

Overview

ggml 1.0.0 prunes the deprecated public API and the legacy DIAG_MASK ops, adds configurable YaRN truncation and FP8 support, and brings broad performance work across backends: few-row quantized matmuls and MoE fusion on CUDA/Metal, a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels, expanded Hexagon op coverage, and scheduler improvements that avoid uploading unused inputs.

API changes

• Pruned deprecated API and removed the legacy DIAG_MASK_INF/DIAG_MASK_ZERO ops, bumping the RPC protocol to reject incompatible peers (llama/29801, llama/31946)
• Added ggml_rope_set_truncate/ggml_rope_get_truncate to control YaRN correction-bound rounding (llama/30206, llama/30802)
• Implemented the backward pass for TRI (llama/30712)
• Added MUL_MAT src1 precision tests and gating for ggml_prec_set_acc (llama/31896, llama/31851)
• Bumped the RPC protocol to v9 with faster loading and graceful failure on unreachable endpoints or failed handshakes (llama/29413, llama/30415)

Core changes

• Scheduler skips uploading inputs that are not read and prefetches only inputs produced by earlier splits (llama/31419, llama/32100)
• MoE cache can compute a share of the cache misses on the CPU and copy the results back asynchronously (llama/30966)
• Meta backend raises the compute view limit for buffers with few tensors (llama/31447)

Backend changes

CUDA

• Added few-row MMQ for batch sizes 2-24 and MMVF for BF16 few-row matmuls (llama/31533, llama/31959)
• Added OCP FP8 support and fused NVFP4 gate/up matmuls with GLU in MMQ (llama/28413, llama/31296)
• Extended FlashAttention with an sm70 head-size-256 config and MMA head sizes 40/72 (llama/27997, llama/30285)
• Improved graph handling: skip compat checks on reuse, re-instantiate on cluster-dim changes, move graph state (llama/32004, llama/31284, llama/32001)
• MoE: skip empty MMQ stream-k tiles, linearize the MoE layout, and sanitize post-bias NaNs (llama/32067, llama/31047, llama/31005)
• Added a debug buffer pool and halved the VMM pool reservation with reserve retries (llama/31919, llama/30542)
• HIP: use a fixed BLAS workspace and add an MFMA lightning indexer for CDNA2 (llama/31838, llama/29050)
• Added a d_state=64 SSM_SCAN fast path and made radix Top-K deterministic for ties (llama/31818, llama/31084)

Metal

• Fused MoE routing and kept fusable node groups inside a single command buffer (llama/31615, llama/31688)
• Made radix Top-K deterministic for ties, added mul_mm for 32<=K<64, and switched TRI to select (llama/31084, llama/30982, llama/30711)
• Fixed GROUP_NORM for ne03>1 and out-of-bounds accesses in the FA output and SOLVE_TRI kernels (llama/31852, llama/31662, llama/32103)

Vulkan

• Added an Intel FlashAttention prefill kernel (llama/29357)
• Added batched tensor copies, a test-fusion interface, and ReBAR detection (llama/31418, llama/30769, llama/31658)
• Enabled dequantized KV with CM2, YaRN truncation, RDNA3/4 large-tile split-K, and large coopmat matmul on newer AMD drivers (llama/31526, llama/30802, llama/30280, llama/31587)
• Added a tiled-transpose concat shader and FWHT kernels for block widths above 512 (llama/30591, llama/29772)
• Fixed MoltenVK out-of-bounds accesses, GROUP_NORM workgroup limits, and FA scratch sizing (llama/31890, llama/31930, llama/30487)
• Capped scalar FlashAttention at 8 rows for large heads on Qualcomm (llama/30829)

SYCL

• Added a device-side MoE scheduler (llama/30156)
• Made the oneDNN FA cache thread-safe and un-gated fused SDPA for Alchemist at head_dim=64 (llama/31138, llama/31574)
• Cached the device id and reject buffers that would exceed VRAM (llama/31126, llama/32013, llama/31285)
• Added exact rounding for CPY to Q4_1/Q5_1/Q8_0, strided CLAMP, and f16 SWIGLU_OAI (llama/31465, llama/31784, llama/30857)
• Fixed AOT for allocations above 4GB (llama/31873)

OpenCL

• Added decomposed Adreno FlashAttention and improved decode performance (llama/32046, llama/31633)
• Added q6_k tiled, q8_0, and f32/q5_k multi-column GEMV kernels and f32 MUL_MAT_ID (llama/31758, llama/30947, llama/31783)
• Added arange, non-contiguous sigmoid, repeat for f16/i32/i16/bf16, and DSV4 HC pre/post kernels (llama/31829, llama/31687, llama/31782, llama/31623)
• Enabled multi-column GEMV by default for Adreno X2 (llama/31694)

WebGPU

• Added batched tensor API support and returned maxBufferSize from get_max_size (llama/31473, llama/31348)
• Added GROUP_NORM, MEAN, COUNT_EQUAL, and PAD_REFLECT_1D kernels (llama/31490, llama/31591, llama/31578, llama/31877)
• Supported the YaRN truncation config (llama/30802)

Hexagon

• Added Q5_0/Q5_1 support and optimized conv_2d while adding conv_3d (llama/32094, llama/31814)
• Fused post-norm residual add into RMS_NORM_MUL and added set_tensor_n (llama/31822, llama/31744)
• Added automatic libhtp loading and fixed 64-bit FastRPC handle truncation (llama/31781, llama/30952)
• Fixed the host build with clang-cl on Windows ARM64 (llama/32011)

CPU

• Switched MUL_MAT to F32 src1 and accumulate FlashAttention V in F32 (llama/31854, llama/29810)
• Added SpacemiT GATED_DELTA_NET RVV and Q4_1 32x256 HP kernels and enabled IME2 MXFP4 repack (llama/31897, llama/31929, llama/31776)
• Enabled the 4x4 simd_gemm tile on ARM NEON and added bf16/q8_0 VXE to llamafile (llama/32128, llama/31866)

CANN

• Fail early when a buffer type is unsupported (llama/31270)

ZDNN

• Accept non-zdnn tensors, invalidate compute tensors after each op, and fix release backend lib detection (llama/31824, llama/31905)

OpenVINO

• Supported the YaRN truncation config (llama/30802)

More info

• Releases and versioning of ggml-org projects

Changelog since v0.29.0

417a024b ggml : bump version to 1.0.0 (#1690) 771b3d9c requirements : fix numpy for python 3.14 (#1691) f154959e sync : llama.cpp ec04faf1 fix(hexagon): make host build work with clang-cl on Windows ARM64 (llama/32011) b45738e9 examples : adapt to removal of ggml_diag_mask_inf 15144390 sync : llama.cpp 5f295554 ggml : prune (llama/29801) 076861cd hexagon: add Q5_0 and Q5_1 support (llama/32094) 63ec52d4 Title: CUDA: few-row MMQ for BS 2-24 (llama/31533) c6fe7968 ggml : route GATED_DELTA_NET to the SpacemiT RVV kernel (llama/31897) 4ae6e721 vulkan : Intel FA kernel for prefill. (llama/29357) b22071c0 ggml-cpu: use the 4x4 simd_gemm tile on ARM NEON (llama/32128) ba8b0130 CUDA: Optimize prefil via fuse of w_s scale in epilogue MMQ for nvfp4 checkpoints (llama/26048) 05f605a2 ggml-backend : prefetch only inputs computed by earlier splits (llama/32100) b8a5b871 ggml : dispatch SpacemiT Q4_1 32x256 through the HP kernel (llama/31929) 7a8cef58 sycl: make oneDNN FA cache thread-safe and memoize runtime failures (llama/31138) 6dfea1a1 sycl : fix AOT for more than 4GB (llama/31873) 012ff44a sycl : use cached device-id in set_main_device (llama/32013) 2f9d47bb CUDA: add cuda debug buffer pool (llama/31919) 522d2575 opencl: add bin kernel kernel_gemm_noshuffle_q4_0_f32_32b_trans_ila_a7_bin (llama/30947) 0f9dcb72 CUDA: skip empty MoE tiles in MMQ stream-k fixup (llama/32067) 6bed9a83 opencl: add decomposed FA for Adreno (llama/32046) a357e7bc metal : prevent out-of-bounds reads in SOLVE_TRI kernel (llama/32103) c36ccd88 vulkan: support batched tensor copy API (llama/31418) 1bffc59d backend : re-instantiate CUDA graphs when the cluster dims of a kernel change (llama/31284) 81531556 ROCm/HIP: fixed BLAS workspace instead of rocBLAS allocating one per call (llama/31838) ad7d200d cpu: accumulate flash attention V in F32 (llama/29810) 7f4a2db3 webgpu: add PAD_REFLECT_1D support (llama/31877) 0f514548 vulkan: allow dequantized KV optimization with CM2 (llama/31526) eb4de491 vulkan: allocate the device fault vendor binary buffer (llama/32008) 6b7f47aa cuda : use MMVF for BF16 matrices of few rows at small batch sizes (llama/31959) 77d01f1b cuda : add sm70 FlashAttention config case for DKQ=256, DV=256, ncols=64 (llama/27997) bd37ba0d cuda : fattn mma head sizes 40 72 (llama/30285) 161ca156 CUDA: Add OCP FP8 support (llama/28413) 06faf806 sycl : fixed sycl CLAMP to actually support strided view (llama/31784) a986181f cuda : remove unused ids and type_src0 parameters (llama/32018) 55104b29 llama : Add FP8 KV-cache support (llama/28392) 0753641f ggml : add MUL_MAT src1 precision tests and gating (llama/31896) 1cd16bd4 cuda : remove unused quantize_cuda_t typedef (llama/31884) 4a18cdd8 cuda : halve VMM pool reservation size and retry on cuMemAddressReserve failure (llama/30542) 1c9ac9d8 CUDA: skip graph compat check when graph is being reused (llama/32004) 999c7250 ggml : fix FreeBSD Vulkan build by setting _XOPEN_SOURCE=700 (llama/31701) 47bc1d84 CUDA: move graph and concurrent stream state out of common.cuh (llama/32001) df70e121 cann: fail early when checking if buffer type is supported (llama/31270) e447eb53 sycl: Add device-side MoE scheduler (llama/30156) 9a1c5822 vulkan: add tiled-transpose concat shader for Qwen models (llama/30591) 8b3c8cf9 ggml-rpc : fail gracefully on unreachable endpoints and failed handshakes (llama/30415) f860f1dd vulkan: Divide GROUP_NORM workgroup to 2D to avoid maxComputeWorkGroupCount limit (llama/31930) eb5571a3 vulkan: Fix out of bounds reads/writes on MoltenVK (llama/31890) 5a9a2643 vulkan: cap scalar flash attention at 8 rows for large heads on Qualcomm (llama/30829) 5610f41b sycl: un-gate oneDNN fused SDPA for Alchemist at head_dim=64 (llama/31574) 487cadf8 hexagon: fix 64-bit FastRPC handle truncation on close (llama/30952) be37c9cf ggml-zdnn: accept non-zdnn tensors, invalidate compute tensors after every op (llama/31824) 294d5fc5 ggml : implement the backward pass for TRI (llama/30712) e18c2c60 ggml-backend: don't upload inputs that are not read (llama/31419) e84d36b5 metal : prevent out-of-bounds writes to the Metal FA output buffer (llama/31662) 0dd184f8 metal : make ggml_tri select instead of scaling by the comparison (llama/30711) b6d482c5 opencl : include climits (llama/31935) 62de7806 RPC: Improve loading times (llama/29413) a7276d0a ggml : support YaRN truncation config on openvino, sycl, vulkan, webgpu, rpc (llama/30802) da894596 sycl: cache device-id (llama/31126) fb570857 tests: bound the test_rope tolerance by the f32 angle resolution (llama/31874) e43d8477 opencl: add f32 MUL_MAT_ID (llama/31783) d1c1e4b2 ggml-zdnn: fix release binary not detecting backend lib (llama/31905) 7e7b6e91 mtmd: refactoring: remove clip.h, consolidate code into mtmd (llama/31865) 5a0e5368 cpu: enable SpacemiT IME2 MXFP4 dense repack (llama/31776) 7f4b25f7 llamafile: impl bf16 and q8_0 vxe (llama/31866) f22fc6a2 ggml : update CUDA/Metal radix Top-K to make tied selections deterministic (llama/31084) 47b8b4fc ggml-cuda : add d_state=64 fast path to SSM_SCAN kernel (llama/31818) b5249606 webgpu : add COUNT_EQUAL support (llama/31578) 75a4f5ab ggml : switch to F32 src1 in the CPU backend (llama/31854) 6cddab2b llama, mtmd: follow the reference precision of ggml_prec_set_acc calls (llama/31851) 7353275d opencl: improve q8_0 and MoE small-n verification for spec (llama/31831) 8f1404ef hexagon: support for automatic loading of libhtp without legacy env vars (llama/31781) 91c102a6 sycl: exact rounding in CPY to Q4_1, Q5_1 and Q8_0 (llama/31465) 8cf32bc6 sycl: refuse device buffers that would exceed VRAM (llama/31285) d969b07e hexagon: n_hvx and n_threads cleanup (llama/31857) 5746ebd6 llama_moe_cache: run missed experts on CPU, add tuning script (llama/30966) a49f174d sycl: add f16 support for SWIGLU_OAI (llama/30857) c6a07809 metal : fix GROUP_NORM for ne03 > 1 and non-contiguous inputs (llama/31852) 1e5c3e03 webgpu : Implement group norm kernel (llama/31490) 37ecdca3 vulkan: Add 'readonly' to decodeBuf declarations. (llama/31833) c08538b3 CUDA: fix shared memory bank conflicts in the lightning indexer (llama/31803) 07b8362f opencl: add arange (llama/31829) f55a5045 meta : raise the compute view limit for buffers with few tensors (llama/31447) 28701331 hexagon: fuse post-norm residual add into RMS_NORM_MUL (llama/31822) a135ec67 hexagon: conv_2d optimizations and support for conv_3d (llama/31814) e3d36332 tests : qualify isnan and signbit with std:: (llama/31791) a7d3330a tests : override hardware_concurrency in CI (llama/31797) ab7ca9c6 opencl: support repeat for f16, i32, i16, bf16 (llama/31782) 9290392e webgpu: add batched tensor API (get_tensor_n and set_tensor_n) (llama/31473) f594fedf vulkan: detect ReBAR support, avoid BAR allocations if unavailable (llama/31658) 38e4be2b CUDA: fix Maxwell build (llama/31787) 8bd6b61b opencl: add q6_k multi column tiled GEMV for speculative verification and refactoring (llama/31758) d3ce63de hexagon: Implement set_tensor_n and improve repacking/tiling robustness (llama/31744) 9b1880f9 ggml-cuda: sanitize post-bias NaNs in MoE selection (llama/31005) 7e607b8f vulkan: Implement test-fusion interface (llama/30769) 2188d132 CUDA: use highest compiled arch in get_mmvq_mmid_max_batch (llama/31679) 114ce5ed vulkan: RDNA3/4 Large-Tile Split-K Optimization (llama/30280) e074347a metal : fuse DeepSeek-V4/V4.1 and GLM MoE routing (llama/31615) f0f5b420 CUDA/HIP: matrix-core (MFMA) lightning indexer for CDNA2 (gfx90a) (llama/29050) fa8c6179 CUDA: linearize MoE data layout for MMQ/MMF (llama/31047) 1690d2e1 vulkan: FWHT kernels for block widths above 512 (llama/29772) 40278c41 vulkan : check combined K+V f16 scratch size in FA dequant path (llama/30487) 2dccfdb2 sync : llama.cpp 25b845e8 ggml : support truncate=false for YaRN (llama/30206) accf2a1f sync : llama.cpp b2ce926c metal : use mul_mm for 32 <= K < 64 (llama/30982) 1a5f4771 webgpu : add MEAN support (llama/31591) 2ab6d6c4 vulkan: Re-enable large coopmat matmul configuration on AMD proprietary driver 26.10.1+ (llama/31587) 19df6cfe tests: fix CI flakes of TOPK_MOE caused by near ties (llama/31426) a0df99d7 metal: keep fusable node groups inside one command buffer (llama/31688) 3a996dba opencl: enable f32 and q5_k multi column GEMV by default for Adreno X2 (llama/31694) d75184eb opencl: support non-contig sigmoid (llama/31687) c6345010 opencl: add DSV4_HC_PRE and DSV4_HC_POST (llama/31623) 08161372 Title: CUDA: fuse NVFP4 gate/up matmuls and GLU in MMQ (llama/31296) 4ac0dae8 opencl: improve decode perf with FA (llama/31633) 4f681f28 webgpu: return maxBufferSize from get_max_size and avoid whole-buffer bindings in memset/clear (llama/31348) cd6d2b75 docs : sync ggml_type enum in gguf.md (#1685)