Why generation needs its own memory
This article examines arXiv version 1 of “Efficient Memory Management for Large Language Model Serving with PagedAttention,” submitted by Woosuk Kwon, Zhuohan Li and colleagues on 12 September 2023. Autoregressive models generate tokens one at a time. Their KV cache retains computed attention keys and values, not a document containing the user’s text. Longer sequences require more cache. Reserving a large contiguous region for each request wastes unfilled space and creates fragmentation when requests have different lengths. PagedAttention addresses allocation and sharing rather than retraining the model.
Logical order is separate from physical placement
The method stores a fixed number of tokens’ KV data in each block. A block table maps the request’s logical sequence to physical GPU blocks and tracks filled positions; neighboring logical blocks need not be physically adjacent. Allocate blocks for the prompt, write subsequent tokens into the last block, and allocate another only when necessary. Attention reads through this mapping, preserving token order. Requests can share a memory pool while retaining their own mappings: unrelated customers’ conversations are not merged.
Two answers to one prompt: a hypothetical example
Suppose a support tool generates answers A and B from exactly the same prompt and instructions. Their tables can point to the same physical input blocks. Generated portions diverge and need separate storage; matching words alone do not imply equal KV values because context and position matter. If a shared final input block still has room, writing a new token requires copy-on-write when other references remain. Filled common blocks stay shared; only the block needing separation is copied, or a new block is allocated. Blocks with no remaining references are reclaimed. The illustration is conceptual, not a product screen or measured result.
What the approximately twofold result means
Section 6.5 reports that vLLM sustained approximately twice the request rate of three Orca baselines in a chatbot experiment. This concerns throughput under the tested latency behavior, not doubled answer accuracy or every answer finishing in half the time. The experiment used synthesized request arrivals and did not keep KV caches between conversation rounds. Its result is not a measurement of the hypothetical support service. Input/output lengths, GPU capacity and traffic patterns affect gains.
Storage management and the implementation record
Block allocation does not retrieve only selected history, discard less important tokens or automatically extend the model’s context window. Smaller blocks reduce unused tail space but introduce mapping overhead. FlashAttention-2 focuses on attention computation and parallel work; PagedAttention focuses on placement and sharing of growing KV data. For development and IP review, record allocation timing, mapping structure, sharing conditions, copying and reclamation, with input/output lengths and concurrent-request measurements. Distinguish these practical recommendations from the paper’s experiments and explain changed steps rather than asserting only “less memory.”
The original figure labels and forms are preserved; the text explains the source material.
PagedAttention paper, version 1
Original technical analysis on Tistory