Dissecting Memory Bandwidth & SRAM Tiling in Triton: From GEMM to FlashAttention-3
Why LLM inference is strictly memory-bound in decoding, how HBM3e vs SRAM bandwidth dictates latency, and writing a fused Triton kernel with online softmax.
H100 SXM5
3.35 TB/s (98%)
Triton 3.0
-93.6% DRAM