ggml Files

Overview

ggml v0.27.0 extends the backend scheduler API with user callbacks for split weight copying and compute-buffer reallocation, and bumps the RPC protocol to v8 to support tensor-parallel RPC servers. Backends gained substantial new fast paths: chunked gated-delta-net kernels on CUDA/HIP, KleidiAI SME2 k-quant matmuls and AVX2 Q2_0 dot products on CPU, L2-aware coopmat2 matmul scheduling and bf16 activations on Vulkan, reordered-layout MoE kernels (MXFP4, IQ3, Q5_K, grouped XMX GEMM) on SYCL, dp4a MoE GEMMs on OpenCL, tensor-API and wider-head-dim flash attention on Metal, and a DMA/HVX copy overhaul with new pooling and 3D im2col ops on Hexagon. Correctness work concentrated on non-contiguous and strided views, grid/workgroup size limits, out-of-bounds accesses, and well-defined mul_mat_id behavior for duplicate expert ids.

API changes

• Add a scheduler copy callback (ggml_backend_sched_set_copy_callback) for user-side copying of split weights (llama/29943)
• Add a scheduler realloc callback (ggml_backend_sched_set_realloc_callback) for compute buffer reallocations (llama/30329)
• Bump RPC_PROTO_MAJOR_VERSION to 8 for the new tensor-parallel RPC server mode (llama/26610)
• Document that ggml_mul_mat_id is undefined for ids that repeat an expert for the same token (llama/30501)
• Document that ggml_gated_delta_net only guarantees the latest state slot when n_tokens > K (llama/29353)
• Default symbol visibility is now hidden, removing internal symbols accidentally exported through GGML_API (llama/30014)

Core changes

• Fix out-of-bounds write in ggml_acc/ggml_set when a negative offset sign-extends past the bounds check (llama/30135)
• Fix ggml_conv_1d result layout for batched input so the output is [N, OC, OL] (llama/28738)
• Make CLAMP correct on non-contiguous views in the CPU and CUDA backends (llama/29517)
• Meta backend: treat views of host-buffer tensors as unsplit and skip them in subgraph mapping (llama/30217)
• Meta backend: recreate all subgraphs when the compute arena is reallocated (llama/30453)
• RPC: make GGML_RPC_DEBUG a verbosity level, add transport logs and clearer tensor transfer logs (llama/29544, llama/30481)

Backend changes

CPU

• Add AVX2 dot product kernel for Q2_0 (llama/30447)
• KleidiAI: add SME2 matmul for Q4_K and route Q6_K to the Q8_0 path (llama/29219)
• Restore the FP16->FP32 lookup table when F16C is available, and fix AVX2 builds not using F16C (llama/30342, llama/30343, llama/30270)
• Vectorize the fp32->fp16 conversion and optimize pad_f32 (llama/30157, llama/30526)
• Add F16 reference implementation and validation for SWIGLU_OAI (llama/29200)
• Size the set_rows conversion buffer for all threads (llama/30454)

CUDA / HIP

• Add chunked gated-delta-net kernels, plus more GDN state columns per warp (llama/29353, llama/30087)
• Support arbitrary striding in unary ops on f16/f32/bf16 and byte-strided ROLL on non-contiguous inputs (llama/29781, llama/29547)
• Improve top-k algorithm selection (llama/28713)
• Add BF16 support for XIELU and chunk BF16/FP16->FP32 conversions (llama/29955, llama/29442)
• FWHT for block widths above 512, SIMT implicit GEMM conv2d without WMMA, better pool2d coalescing (llama/29100, llama/30325, llama/29425)
• Prefer MMVQ for MUL_MAT_ID on sm_60 and pass src1 precision to host MMQ config helpers (llama/27828, llama/30168)
• Fix MMQ out-of-bounds reads and alignment assumptions when bias-add is fused (llama/29953, llama/30133)
• Handle tensors beyond grid dimension limits in the PAD and norm family kernels (llama/30147, llama/28175)
• Fix flash attention race when combining partial results and drop redundant copies after SSM_SCAN (llama/30103, llama/29807)
• Fix the CCCL version guard on major version rollover and CPU/GPU rounding agreement under MSVC (llama/29453, llama/30229)
• Add GGML_CUDA_GRAPH_DEBUG, read backend env vars once, and use per-thread streams for buffer-init memset (llama/30327, llama/28782)
• HIP: 64-row MMQ tiles for selected quant types on RDNA3.5 (llama/28195)
• MUSA: use the tile lightning indexer kernel, fix FWHT, drop an obsolete default architecture (llama/30080, llama/30167, llama/30203)

Metal

• Add a BF16 tensor-API flash attention kernel and new 128/96 head-dim FA kernels combinable across dtypes (llama/30491, llama/30209)
• Extend the few-row MMA mat-mul to the remaining src0 types and fix tensor-API mul_mm on large tensors (llama/30065, llama/28748)
• Fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT, and excess threadgroup memory in quantized FA (llama/30100, llama/29340)
• Fix mul_mat_id with duplicate expert ids (llama/30373)

Vulkan

• L2-friendly workgroup scheduling for coopmat2 matmuls, and dispatch splitting beyond workgroup count limits (llama/30215, llama/29533)
• Support bf16 activations in mul_mat and extend sparse flash attention to coopmat2 (llama/29069, llama/30003)
• Handle duplicate mul_mat_id ids in the prepass instead of looping, and use 4 rows for NVIDIA MUL_MAT_ID MMVQ (llama/29998, llama/29274)
• Overhaul host-visible memory so several devices can share it, and fix concurrent first-device initialization (llama/29741, llama/29766)
• Copy strided f16 KV once on AMD so reads spread across memory channels (llama/27703)
• Pack FMAs for f16 accumulation without the dot product op, and speed up large-window pool_2d (llama/29877, llama/30119)
• Fixes: TOP_K with +inf/NaN and k = 1 on negatives, rms_norm workgroup overflow, slow AMD iGPU reads (llama/30107, llama/30145, llama/30049)
• Intel GPU workarounds: handle free memory reporting above total memory, set shader counts for Arc Pro B65/B70 (llama/29835, llama/30431)

SYCL

• Accelerate MoE with arithmetic decoding and weight reordering for MXFP4, IQ3 and Q5_K reorder layouts (llama/29809, llama/29107, llama/29500, llama/29375)
• Add grouped MoE XMX GEMM and fuse the delta-net alpha gate (add + unary + mul) (llama/29245, llama/29687)
• Accelerate MLA prefill with MKL flash attention and fix flash attention across heterogeneous GPUs (llama/29171, llama/29071)
• Stage model-loading uploads through a pinned host ring buffer (llama/29608)
• Optimize FWHT and guard DMMV reads at row tails (llama/29605, llama/29338)

OpenCL

• Add dp4a MoE binary GEMM kernels for Q4_K and Q6_K against Q8_1 (llama/30187)
• Flash attention improvements: apply softcap before the mask, widen supported head dims (llama/30546, llama/30266)
• Support CLAMP on non-contiguous views (llama/30186)
• Adreno: fix out-of-bounds tail access in noshuffle GEMV and out-of-bounds read in xmem GEMM (llama/30421, llama/30041)
• Adreno: fix q8_0 dp4a mul_mat_id, mark PREC_F32 mul_mat_id unsupported, sort experts without sub-buffers (llama/30365, llama/30418)
• Fix image size limits in dense binary kernels and kernel compilation for older GPU compilers (llama/30310, llama/30176)

WebGPU

• Use 2D workgroup dispatch for ops that previously used 1D dispatch (llama/30219)
• Add DUP op support and skip Dawn native features on WASI (llama/30216, llama/27069)
• Fix mul_mat_id with duplicate expert ids and flash attention support detection for overlapping KV (llama/30451, llama/28205)

Hexagon

• Add configurable HTP DCVS/HMX power modes via GGML_HEXAGON_POWER_MODE (llama/30423)
• Overhaul CPY/CONCAT/CONT/DUP onto DMA and HVX, including short-row and scalar broadcasting paths (llama/30067, llama/30341, llama/30245)
• Add IM2COL_3D and pooling ops, and fix the patch-embed DMA ring overflow (llama/30305, llama/29995, llama/30189)
• Faster Q2_K/Q6_K dequant and activation, tiled Q4_K/Q6_K GET_ROWS, better GELU accuracy (llama/30035, llama/30121, llama/30115, llama/30104, llama/29971, llama/29974)
• Enable the multi-buffer allocation API (llama/30126)

zDNN

• Add Q8_0 matrix multiplication support (llama/29836)

OpenVINO

• Fix stateful GPU execution and op-table regressions, add DUP and rank-alignment handling (llama/30037)

RPC

• Add tensor-parallel server mode with async graph compute, custom all-reduce and 2D tensor transfer (llama/26610)

More info

• Releases and versioning of ggml-org projects

Changelog since v0.26.0

a05c9922 ggml : bump version to 0.27.0 (#1659) 524441cb sync : llama.cpp 00fb915e vulkan : fix concurrent first device init (llama/29766) 3db777ac opencl: fix OOB tail access for adreno noshuffle gemv kernels (llama/30421) f88e4141 ggml-cpu : optimize pad_f32 (llama/30526) b5c29f19 opencl: apply flash attn softcap before mask (llama/30546) 0a38c7db vulkan : copy strided f16 KV once on AMD so reads spread across memory channels (llama/27703) c366bd97 cpu: add F16 SWIGLU_OAI reference implementation and validation (llama/29200) 69d79128 metal : add BF16 variant of the tensor API flash attention kernel (llama/30491) 70a99665 vulkan: Use L2-friendly workgroup scheduling for cm2 mulmat (llama/30215) 79723b4b opencl: support CLAMP on non-contiguous views (llama/30186) c1209bf2 sycl: IQ3 code reorder (llama/29107) 41f2ae65 ggml-cpu : keep the FP16->FP32 lookup table on x86 with F16C (llama/30342) (llama/30343) 73d50f0e ggml-cpu : size set_rows conversion buffer for all threads (llama/30454) 1f344416 cuda, hip: GDN chunked kernel (llama/29353) cb7386da ggml: fix conv_1d result layout for batched input (llama/28738) e990bfbc ggml : add note about mul_mat_id + disable mul_mat_id_dup tests (llama/30501) bf18f8a0 ggml : improve RPC tensor transfer logs (llama/30481) 0564e931 cuda : use SIMT implicit GEMM conv2d on AMD without WMMA (llama/30325) a6e3198c opencl: use buf+offset instead of subbuffer for adreno moe expert sort (llama/30418) d4a2ebaa opencl: fix q8_0 dp4a mul_mat_id and mark PREC_F32 mul_mat_id unsupported for adreno (llama/30365) 8d2974e9 HIP: use 64-row MMQ tiles for some quant types on RDNA3.5 (llama/28195) d9936e67 vulkan: fix compile error (llama/30495) 65c60b23 vulkan: Add workaround for device free memory > total memory on Intel Windows (llama/29835) bc77a961 vulkan: Define the shader count for Intel Arc Pro B65 + B70 (llama/30431) 3cbffc97 vulkan: support mulmat bf16 activations (llama/29069) 433c1279 webgpu: (Fix CI) fix mul_mat_id with duplicate expert ids (llama/30451) 3879eb0f ggml : simplify backend search path error handling (llama/28367) 02ed0947 kleidiai: add SME2 Q4_K and Q6_K matmul support (llama/29219) 9e72ba90 vulkan: Optimize pool_2d when the pooling window is large (llama/30119) 3eb7b78f meta: recreate all subgraphs when the arena is reallocated (llama/30453) 8c098a0f hexagon: add configurable HTP DCVS/HMX power mode (llama/30423) 442a0bb8 tests : consolidate context state tests into test-llama-context (llama/30023) fae97b3f vulkan: allow multiple devices to access gpu-visible host memory (llama/29741) 47e7d28e ggml-cpu : add AVX2 dot product for Q2_0 (llama/30447) 73a8ca2c cuda : add GGML_CUDA_GRAPH_DEBUG and read env vars once (llama/30327) f2e27b61 cuda : fix compiler warnings in mmq-load-tiles.cuh (llama/30443) 49f18d18 vulkan : split matmul dispatch when it would exceed maxComputeWorkGroupCount (llama/29533) 1955eed9 cuda : fix warnings from dead breaks (llama/30441) 07f194d2 cuda : warnings cleanup (llama/30420) 40e618b1 ggml-zdnn : introduce Q8_0 mat mul (llama/29836) fa129846 hexagon: extend scalar DMA broadcasting support for binary ops (llama/30245) 3865c11a metal: fix metal tensor kernel_mul_mm error for large tensor (llama/28748) bdaa8135 hexagon: optimize CPY/CONCAT — round 3, short-rows edition (#30341) db45afad vulkan: pack FMA for f16 accumulation, no dot product op (llama/29877) 919a0bb0 metal : fix mul_mat_id with duplicate ids (llama/30373) 876ecfa2 llama : report compute graph reallocations during inference (llama/30329) 1ad98140 sycl: guard DMMV reads at row tails (llama/29338) 687faba6 vulkan: use ternary expressions to avoid switch lowering for CM2 quant block size (llama/30164) 1308ac7b ggml: Change default visibility of symbols to hidden (llama/30014) 1c5e9ac4 vulkan: fix buffer logging on 32-bit platforms after source split (llama/30326) bd19f7e0 hexagon: optimize q2_k bpw and activation (llama/30035) 49c0a25a hexagon: add IM2COL_3D (llama/30305) d1d14004 opencl: improve fa, allow dk512 for gemma-4, improve dk64 (llama/30266) a7e9ac52 opencl: fix image limit for q4_k dense bin kernels, q4_0 bcast and rms_norm (llama/30310) 03360098 rpc : turn GGML_RPC_DEBUG into a verbosity level and add logs (llama/29544) 2505f988 cpu: fix AVX2 CPUs not using F16C when available (llama/30270) c098ee22 vulkan: handle mul_mat_id duplicates in the prepass rather than looping (llama/29998) 4ed80dc4 cmake(s390x): disable z17 target for unsupported compilers (llama/30297) ecb19c6d cuda: always use MMVQ for MUL_MAT_ID on sm_60 (llama/27828) 9c3d610a CUDA/HIP: fix race in flash_attn_ext_f16_process_tile when nbatch_combine != DKQ/2 (llama/30103) 0ee2b4cc ggml-cpu: vectorize fp32 to fp16 conversion (llama/30157) 678fb23d vulkan: use 4 rows for NVIDIA MUL_MAT_ID MMVQ (llama/29274) 98895378 ggml: fix s390x all cpu build (llama/30140) ee26a0a1 opencl: add bin kernels kernel_gemm_moe_q4_k_q8_1_dp4a_bin, kernel_gemm_moe_q6_k_q8_1_dp4a_bin (llama/30187) 1540062a sycl : accelerate MXFP4 MoE with arithmetic decoding and weight reordering (llama/29809) e66c2c4c CUDA: fix round issue, under MSVC the CPU and GPU agree (llama/30229) 40a76a80 CUDA: Remove redundant CUDA copies after SSM_SCAN (llama/29807) 7e03c292 opencl: fix kernel compilation for a6x GPUs (llama/30176) 90839562 ggml: fix OOB write in ggml_acc with negative offset (llama/30135) abb056e4 meta : handle host views (llama/30217) eac27d87 webgpu: use 2D workgroup dispatch for all the ops which use 1D dispatch (e.g., rms_norm) (llama/30219) bdda9e2d vulkan: fix rms_norm workgroup count overflow (llama/30145) 49f72e3f musa: drop mp_21 from the default architectures (llama/30203) 2bccca94 ci: fix Models Backend webgpu by supporting GGML_OP_DUP (llama/30216) 921e4594 metal: add the 128/96 flash attention kernels, accept every compiled pair (llama/30209) 1160710c CUDA: pass src1 precision to host MMQ config helpers (llama/30168) 682f479f hexagon: fix IM2COL patch-embed DMA ring overflow (llama/30189) e2a654c0 sycl : Q5_K reorder-layout MMVQ and fused GLU (llama/29375) 95ab266f Musa FWHT fix (llama/30167) 315f00fb CUDA: improve top-k algorithm selection (llama/28713) e02d4bfc CUDA: fix MMQ out-of-bounds reads (llama/29953) 03f0267b CUDA : looped PAD kernel for more than 65535 rows or slices (llama/30147) edd2b43d CUDA: fix CCCL version guard breaking on major version rollover (llama/29453) 8bc17888 vulkan : fix TOP_K for +inf/NaN inputs and k = 1 on negative values (llama/30107) 3c174213 vulkan: extend sparse FA support to coopmat2 (llama/30003) cd37cae6 cuda : support arbitrary striding for unary ops on f16, f32, and bf16 (llama/29781) e6cabf61 sycl: FWHT optimizations (llama/29605) 94386c8c hex-mmadd: do not assume aligned read/write when bias-add is fused (llama/30133) d48b5724 cuda : Use byte strides for roll to allow non-contiguous ROLL operations (llama/29547) 757416a0 CUDA: fix norm family kernels when ne[2]/ne[3] exceed grid dim limits (llama/28175) 1c0fbee2 sycl: fuse the delta-net alpha gate (add + unary + mul) (llama/29687) 217561da sycl: stage bulk uploads (model loading) through a pinned ring buffer (llama/29608) 5e14831c ggml-cuda: assign four GDN state columns per warp (llama/30087) 1697602b sycl: add grouped MoE XMX GEMM (llama/29245) 34933fc8 sycl: remove duplicate block-size defines from op headers (llama/29507) 139ba6f2 hexagon: enable alloc_buffer_n (llama/30126) 3ad83a62 hexagon: Q6_K weight dequant speedup (llama/30121) 55d17818 hexagon: support tiled Q4_K and Q6_K GET_ROWS (llama/30115) 1510e16c hexagon: improved GELU accuracy (llama/30104) 46b8d58b cuda: FWHT kernels for block widths above 512 (llama/29100) cc642d69 cuda: update uncoalesced memory reads in pool2d (llama/29425) 871ed068 metal : few-row MMA mat-mul for the remaining src0 types (llama/30065) d7ae7d98 metal : fix MUL_MAT+ADD fusion when the residual is itself a MUL_MAT (llama/30100) 86ce6fe1 musa: use the tile lightning indexer kernel (llama/30080) 9dd84f71 ggml-webgpu: fix flash_attn supports_op check for overlapping KV (llama/28205) c5a49eb9 fix the issue in mixed different model GPUs in FA (llama/29071) 03bb3518 sycl: accelerate GLM MLA prefill with MKL flash attention (llama/29171) e1bf4673 sycl : fattn_kv_buffers cleanup (llama/27689) 4f73a3ff vulkan: fix amd iGPU slow checkpoint read (llama/30049) ce696e0a sycl: add IQ3_S multi-column MMVQ (llama/29500) 44878ef9 ggml-webgpu: no dawn native features on wasi (llama/27069) 57ca9161 sync : llama.cpp 7758af5d hexagon: CPY/CONCAT/CONT/DUP overhaul to use DMA/HVX for all cases (llama/30067) b5835d59 sync : llama.cpp 196fc955 cuda : add BF16 support for XIELU (llama/29955) ee665f42 ggml-cuda: use per-thread stream for buffer-init padding memset (llama/28782) 62be868f opencl: fix OOB read in adreno xmem GEMM (llama/30041) cf98044c RPC: add -sm tensor (llama/26610) e5d6d3b8 ggml: fix CLAMP on non-contiguous views (CPU, CUDA) (llama/29517) 9b4e0375 cuda: BF16/FP16 conversion to f32 chunking (llama/29442) d1e01153 metal : fix excess threadgroup memory in quantized flash attention (llama/29340) 5faddf60 vulkan : check for null vkEnumerateInstanceVersion (llama/29872) afe2a924 HIP: use -O0 for host code in debug builds (llama/29795) 5558a3be ggml: refactor selective expert copying to user code (llama/29943) dd77bafe ggml-openvino: fix CI tests; fix GPU regressions. (llama/30037) 46f70991 hexagon: ssm-conv updates (llama/29971) ca609a85 hexagon: add pool op support (llama/29995) 0dfa55eb hexagon: matmul and flash-atten scalability updates (llama/29974) 3612c0af sam : remove redundant layout copies in the image encoder and mask head (#1654) aaf967f7 scripts : add apiabi release check (#1653) ffa4e8b8 scripts : update release summary prompt