b12543
1 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · chưa có số tương tác · Nguồn crawl
Chi tiết nguồn
<details open=""> <p>metal : CONV_2D tensor API for M5, split-K for all chips (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="6468918084" data-permission-text="Title is private" data-url="https://github.com/ggml-org/llama.cpp/issues/32542" data-hovercard-type="pull_request" data-hovercard-url="/ggml-org/llama.cpp/pull/32542/hovercard" href="https://github.com/ggml-org/llama.cpp/pull/32542">#32542</a>)</p> <ul> <li>metal: add a tensor API conv_2d kernel and split long reductions</li> </ul> <p>The conv_2d matmul kernel runs on simdgroup matrices only, and gives<br> each threadgroup the whole reduction of its output tile. On M5 this<br> leaves the tensor cores unused, and convolutions with few output<br> positions and a long reduction (IC * KW * KH in the thousands) run a<br> handful of threadgroups on an otherwise idle GPU.</p> <p>Add kernel_conv_2d_mm_tensor for devices with the tensor API: like<br> kernel_mul_mm, it gathers the patches of 64 output positions into<br> threadgroup memory as the left operand and reads the kernel rows<br> from device memory as the right operand. Devices without the tensor<br> API keep the simdgroup kernel.</p> <p>Both kernels can now split the reduction until there are enough<br> threadgroups: each split writes its partial sums to a buffer placed<br> after the output, as flash attention does, and a small kernel adds<br> them in a fixed order, so the result stays deterministic. CONV_2D is<br> marked as an op whose allocation may exceed its output size.</p> <p>test-backend-ops -o CONV_2D passes, with and without<br> GGML_METAL_TENSOR_DISABLE.</p> <p>CONV_2D, F16 kernel, us/run on an M5 Max:<br> [19,19,256,16] k [4,4,256,4096] 19008 -> 5235<br> [16,16,128,8] k [3,3,128,512] 333 -> 124<br> [1536,1536,32,1] k [3,3,32,64] 12086 -> 6512<br> [33,1,1280,1] k [3,1,1280,1280] s2 659 -> 33<br> [2054,1,768,1] k [7,1,768,768] 3161 -> 699</p> <ul> <li>metal: walk the conv_2d patches incrementally, size splits by output</li> </ul> <p>The patch gather of both conv_2d matmul kernels spent most of its<br> time on address math: 64-bit multiplies, divisions and four bound<br> checks per value. Walk the reduction index with 32-bit byte offsets<br> inside the image, updated by additions, with one unsigned compare per<br> bound; the host checks that an image fits. The tensor kernel no<br> longer gathers the reduction indices past the end of its split.</p> <p>The number of splits now targets about 1M partial sums, with at least<br> 3 reduction steps per split, instead of counting the tiles of one<br> kernel.</p> <p>CONV_2D, F16 kernel, us/run on an M5 Max:<br> [19,19,256,16] k [4,4,256,4096] 4998 -> 4512<br> [1536,1536,32,1] k [3,3,32,64] 6721 -> 5354<br> [4096,1,1024,1] k [7,1,1024,1024] 2701 -> 2236<br> [2054,1,768,1] k [7,1,768,768] 1050 -> 871<br> with GGML_METAL_TENSOR_DISABLE:<br> [4096,1,1024,1] k [7,1,1024,1024] 9047 -> 8283</p> <ul> <li>nit: share the CONV_2D simdgroup count between host and kernel</li> </ul> <p>kernel_conv_2d_mm and its pipeline both hardcoded 4 simdgroups per<br> threadgroup. N_SG_CONV_2D in ggml-metal-impl.h now feeds the kernel<br> constant and res.nsg, so the dispatch and the kernel cannot drift apart.</p> <ul> <li>metal: fall back to the generic conv_2d kernel for images past 4 GiB</li> </ul> <p>The matmul conv_2d kernels walk the patches with 32-bit byte offsets<br> inside an image. Instead of asserting on nb13, compute the largest<br> offset actually read from ne/nb, which also covers strided views of<br> the input, and route images that do not fit to the generic kernel.</p> <ul> <li>cont : comments</li> </ul> <p>Co-authored-by: Georgi Gerganov <a href="mailto:ggerganov@gmail.com">ggerganov@gmail.com</a></p> <hr> <p>Co-authored-by: Georgi Gerganov <a href="mailto:ggerganov@gmail.com">ggerganov@gmail.com</a></p> </details> <p><strong>Website:</strong></p> <ul> <li><a href="https://llama.app" rel="nofollow">https://llama.app</a></li> </ul> <p><strong>Attestations:</strong></p> <ul> <li><a href="https://github.com/ggml-org/llama.cpp/attestations/69742362">https://github.com/ggml-org/llama.cpp/attestations/69742362</a></li> </ul> <p><strong>macOS/iOS:</strong></p> <ul> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-macos-arm64.tar.gz">macOS Apple Silicon (arm64)</a></li> <li>macOS Apple Silicon (arm64, KleidiAI enabled) <a href="https://github.com/ggml-org/llama.cpp/pull/23780" data-hovercard-type="pull_request" data-hovercard-url="/ggml-org/llama.cpp/pull/23780/hovercard">DISABLED</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-macos-x64.tar.gz">macOS Intel (x64)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-xcframework.zip">iOS XCFramework</a></li> </ul> <p><strong>Linux:</strong></p> <ul> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-x64.tar.gz">Ubuntu x64 (CPU)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-arm64.tar.gz">Ubuntu arm64 (CPU)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-s390x.tar.gz">Ubuntu 24.04 s390x (CPU)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-26.04-s390x.tar.gz">Ubuntu 26.04 s390x (CPU)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-zdnn-s390x.tar.gz">Ubuntu 26.04 s390x (zDNN)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-vulkan-x64.tar.gz">Ubuntu x64 (Vulkan)</a></li> <li><a href="https://github.com/ggml-org/llama.cpp/releases/download/b12543/llama-b12543-bin-ubuntu-vulkan-arm64.tar.gz">Ubuntu arm64 (Vulkan)</a></li> <li><a href="https://github.com
- b12543 ggml-org/llama.cpp releases · 19 thg 11, 2026