# FlashAttention-2: making long-context attention run efficiently on GPUs A practical reading of Tri Dao’s paper: tiled attention, reduced non-matrix work and better GPU work partitioning, with kernel benchmarks distinguished from full-model training. Source: https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning HOME / NEWS & INSIGHTS NEWS & INSIGHTS FlashAttention-2: making long-context attention run efficiently on GPUs A practical reading of Tri Dao’s paper: tiled attention, reduced non-matrix work and better GPU work partitioning, with kernel benchmarks distinguished from full-model training. AI & Software 2026.10.07 published IPLEX 5 min read Conceptual view of FlashAttention-2 computation and GPU work partitioning Why long inputs put pressure on memory Tri Dao’s FlashAttention-2 paper, first submitted to arXiv on 17 July 2023, concerns how attention is executed on a GPU. Tokens are the units into which a model divides text. Attention compares their representations and weights the information each contributes. A conventional implementation materialises large score and probability matrices; as the sequence grows, both storage and data transfers become costly. Consider a fictional customer-service system asked whether the delivery address of an order already shipped can be changed. It must relate information distributed across a long conversation. The example illustrates computation, not an experiment reported in the paper or a claim about a particular deployed service. A GPU’s arithmetic capacity is of limited benefit if its processing units spend too much time waiting for data. Fictional customer-service example illustrating blockwise processing of a long conversation Tiling preserves the attention calculation The original FlashAttention already divided inputs into tiles, brought them into fast on-chip memory and avoided storing the full intermediate matrices in high-bandwidth memory. Online softmax maintains and rescales running statistics and partial outputs so that blocks contribute to a globally normalised result. Independently normalising each block and simply adding the results would not perform the same calculation. In the customer-service analogy, small groups of notes are handled in a limited workspace while a running summary is updated. The model actually operates on numerical token representations, not the human judgments drawn in the illustration. This is an exact-attention algorithm rather than a sparse approximation, but changing floating-point operations can produce small numerical differences. Output and gradient comparisons therefore use tolerances, not a promise of bitwise identity. Three changes improve GPU utilisation First, FlashAttention-2 reduces non-matrix-multiplication work, including rescaling and parts of the bounds and causal-mask handling, to better exploit the GPU’s fast matrix hardware. Second, it distributes work along the sequence dimension as well as across batches and attention heads. This helps keep more GPU processing units occupied when long sequences leave a small batch size. Third, the forward pass partitions Q, the query representation, among warps while sharing K and V, the keys and values. Each warp produces its own part of the output, reducing exchanges and reductions of intermediate results through shared memory. These changes concern scheduling and data movement as much as the attention formula itself. Read the reported speed-ups with their conditions The author reports roughly a twofold improvement over FlashAttention in attention-kernel benchmarks on an A100 80GB SXM4. The tests vary sequence lengths from 512 to about 16k, head dimensions of 64 and 128, and causal masking, while holding the total batch token count at 16k. Those measurements isolate the attention computation. Separate GPT training experiments on eight A100 80GB SXM4 GPUs report up to about 1.3× end-to-end training speed, using 1.3B- and 2.7B-parameter models and context lengths of about 2k and 8k. Other computation and communication remain in the full workload. Neither result guarantees a twofold improvement in a chatbot’s response time on different hardware. Attention arithmetic remains quadratic in sequence length, and lower memory use does not itself establish better comprehension. Document the implementation difference Evaluate the same model and inputs while recording GPU type, sequence length, batch size, memory consumption and elapsed time. Separate the initial processing of a long prompt from token-by-token generation: prefill and decoding can have different bottlenecks. The original figures accompany this explanation as conceptual illustrations. For a technical review, describe which intermediate matrices are no longer stored, where work is partitioned and how communication is reduced. Flow diagrams and controlled comparisons make those differences assessable. IPLEX reviews the concrete implementation and its effects when analysing AI inventions; a faster benchmark alone is not a conclusion on patentability. Read the original Tistory article Explore our AI patent practice Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE Why long inputs put pressure on memory Tiling preserves the attention calculation Three changes improve GPU utilisation Read the reported speed-ups with their conditions Document the implementation difference TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer OEM exports and proof of trade mark use in Korea ↗ Older Patent against patent: when a Korean scope-confirmation trial is admissible ↗ Related insights AI 2026.10.15 PaTH Attention: encoding the path between tokens An analysis of input-dependent Householder transformations, state tracking and the measured benefits and limits of PaTH Attention. ↗ Read article AI 2026.10.14 Huawei Pangu Weather: linking forecast models across time intervals A technical reading of Huawei’s US20250258318A1 publication: selecting forecast intervals, passing predicted states and combining weather data. ↗ Read article AI 2026.10.13 LaCT: updating an AI model’s temporary memory in large chunks A technical review of Large Chunk Test-Time Training, its fast weights, GPU efficiency, causal processing and task-specific experimental limits. ↗ Read article - https://www.iplexlaw.co.kr/en - https://www.iplexlaw.co.kr/en/blog/category/ai - https://iplexlaw.tistory.com/7 - https://www.iplexlaw.co.kr/en/ai-patents - https://www.iplexlaw.co.kr/ko/blog/flashattention-2-gpu-work-partitioning - https://www.iplexlaw.co.kr/en/contact - https://www.iplexlaw.co.kr/en/blog - https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning#case-flash-en-0 - https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning#case-flash-en-1 - https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning#case-flash-en-2 - https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning#case-flash-en-3 - https://www.iplexlaw.co.kr/en/blog/flashattention-2-gpu-work-partitioning#case-flash-en-4 - https://www.iplexlaw.co.kr/en/contact - https://www.iplexlaw.co.kr/en/blog/brunch-case-1805 - https://www.iplexlaw.co.kr/en/blog/brunch-case-1804 - https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations - https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations - https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models - https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models - https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training - https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training