Why long inputs put pressure on memory
Tri Dao’s FlashAttention-2 paper, first submitted to arXiv on 17 July 2023, concerns how attention is executed on a GPU. Tokens are the units into which a model divides text. Attention compares their representations and weights the information each contributes. A conventional implementation materialises large score and probability matrices; as the sequence grows, both storage and data transfers become costly.
Consider a fictional customer-service system asked whether the delivery address of an order already shipped can be changed. It must relate information distributed across a long conversation. The example illustrates computation, not an experiment reported in the paper or a claim about a particular deployed service. A GPU’s arithmetic capacity is of limited benefit if its processing units spend too much time waiting for data.
Tiling preserves the attention calculation
The original FlashAttention already divided inputs into tiles, brought them into fast on-chip memory and avoided storing the full intermediate matrices in high-bandwidth memory. Online softmax maintains and rescales running statistics and partial outputs so that blocks contribute to a globally normalised result. Independently normalising each block and simply adding the results would not perform the same calculation.
In the customer-service analogy, small groups of notes are handled in a limited workspace while a running summary is updated. The model actually operates on numerical token representations, not the human judgments drawn in the illustration. This is an exact-attention algorithm rather than a sparse approximation, but changing floating-point operations can produce small numerical differences. Output and gradient comparisons therefore use tolerances, not a promise of bitwise identity.
Three changes improve GPU utilisation
First, FlashAttention-2 reduces non-matrix-multiplication work, including rescaling and parts of the bounds and causal-mask handling, to better exploit the GPU’s fast matrix hardware. Second, it distributes work along the sequence dimension as well as across batches and attention heads. This helps keep more GPU processing units occupied when long sequences leave a small batch size.
Third, the forward pass partitions Q, the query representation, among warps while sharing K and V, the keys and values. Each warp produces its own part of the output, reducing exchanges and reductions of intermediate results through shared memory. These changes concern scheduling and data movement as much as the attention formula itself.
Read the reported speed-ups with their conditions
The author reports roughly a twofold improvement over FlashAttention in attention-kernel benchmarks on an A100 80GB SXM4. The tests vary sequence lengths from 512 to about 16k, head dimensions of 64 and 128, and causal masking, while holding the total batch token count at 16k. Those measurements isolate the attention computation.
Separate GPT training experiments on eight A100 80GB SXM4 GPUs report up to about 1.3× end-to-end training speed, using 1.3B- and 2.7B-parameter models and context lengths of about 2k and 8k. Other computation and communication remain in the full workload. Neither result guarantees a twofold improvement in a chatbot’s response time on different hardware. Attention arithmetic remains quadratic in sequence length, and lower memory use does not itself establish better comprehension.
Document the implementation difference
Evaluate the same model and inputs while recording GPU type, sequence length, batch size, memory consumption and elapsed time. Separate the initial processing of a long prompt from token-by-token generation: prefill and decoding can have different bottlenecks. The original figures accompany this explanation as conceptual illustrations.
For a technical review, describe which intermediate matrices are no longer stored, where work is partitioned and how communication is reduced. Flow diagrams and controlled comparisons make those differences assessable. IPLEX reviews the concrete implementation and its effects when analysing AI inventions; a faster benchmark alone is not a conclusion on patentability.
Read the original Tistory article
Explore our AI patent practice