Paper and scope of this analysis
Korean patent attorney Yongdeok Kim examines Learning to Evict from Key-Value Cache by Luca Moschella, Laura Manduchi and Ozan Sener of Apple. The analysis uses the ICML 2026 proceedings version: Proceedings of the 43rd International Conference on Machine Learning, PMLR 306:90470–90496. Kim is the author of this commentary, not an author of the research paper.
The arXiv identifier is 2602.10238, first posted on 10 February 2026 and revised as v2 on 26 June 2026. DOI 10.48550/arXiv.2602.10238 identifies the arXiv record; it is not presented here as a separate PMLR DOI. The central idea of KVP is to learn a retention ranking for each KV cache, favouring information likely to matter later across different memory budgets.
The memory problem
A Transformer stores keys and values for previously processed tokens so that it need not recompute them for every new token. This KV cache grows with context length. Reducing it can also remove a clue needed by a later question. Like reference documents spread over a small desk, the most recently used material is not necessarily the material needed next.
Existing eviction methods use position, past attention or properties of key vectors. KVP instead learns to predict future usefulness. Figure 1 compares highlighted tokens in a travel conversation under KVP, an oracle based on actual future attention, and StreamingLLM. The oracle is a retrospective training reference; the deployed policy does not see future questions.
Learn an ordering, then choose a budget
KVP scores all cached tokens and retains the highest-ranked b entries for a budget of b. The same ordering can serve, for example, budgets of 500 or 1,000 entries. Its objective aggregates the future-attention importance discarded across budgets, rather than learning a separate keep-or-delete rule for a single capacity.
Under the paper’s additive utility definition, sorting by future importance yields a common optimal ordering for the proxy objective. This is not a proof that the ordering maximises answer accuracy for every question. Interactions between tokens and quality of generated answers are not completely captured by adding attention values.
A small policy for each layer and head
The policy receives a token’s key vector, value vector and original position. At deployment it needs neither a query vector nor the complete attention matrix. Each layer and KV head has its own small multilayer perceptron. In the main Qwen2.5-7B setup, 28 layers and four KV heads per layer give 112 policies, each with approximately 650,000 parameters.
Inference reads the current K, V and position features, computes scores, keeps the highest-ranked entries allowed by the budget, and continues attention using the retained cache. The evaluated implementation always retains the first four and last sixteen tokens. Cache capacity is allocated equally across heads; learning unequal per-head budgets is not part of this experiment.
Offline training with future-attention supervision
First, the base model is run to collect query, key and value traces. Later queries attending to earlier keys provide a measure of future utility. These traces allow rewards to be computed without rerunning the full language model at each policy update. The initial trace-collection cost still exists.
The scores define a Plackett–Luce distribution over rankings. Adding Gumbel noise and sorting produces candidate rankings in parallel. REINFORCE updates the policies using a leave-one-out reward baseline, RLOO, based on the other sampled rankings. The reward covers losses across cache budgets, so training addresses small as well as large capacities. Future information supplies offline targets; deployed decisions use only information already cached.
How the experiments were run
The main model is Qwen2.5-7B, with policies trained on roughly 4K-token RULER sequences or OASST2 conversations. Comparators include StreamingLLM, TOVA, SnapKV, LagKV, KeyDiff and K-Norm; the appendix also reports Phi-4-14B results. Different training distributions are evaluated rather than assumed interchangeable.
Short-context experiments compress the cache once after prefill, the initial processing of the input, before generating a response. Long inputs are processed in chunks with previously retained entries. Some short-context tests emulate eviction using attention masks, whereas long-context tests physically compact KV tensors. Accuracy gains in these settings therefore cannot be converted directly into universal memory or latency savings.
Short-context results: read each axis carefully
In Figure 2, Cache Size is the number of retained tokens. Higher RULER accuracy is better, whereas lower OASST2 perplexity is better. KVP performs strongly across a broad range of capacities, and methods tend to converge when ample cache is available.
On OASST2, LagKV can perform better at extremely small budgets. The evidence supports a favourable quality–capacity trade-off in the tested settings, not a claim that KVP wins at every capacity or on every task.
From 4K training to 128K input
Figure 3 applies a policy trained on about 4K tokens to RULER-128K. KVP remains ahead of the plotted baselines across the displayed budgets, suggesting that the learned ranking can transfer to much longer inputs.
Absolute performance still matters: at a cache size of 10,000, the plotted accuracy is approximately 29%. This is an approximate reading of the chart, not perfect recall of a 128K document. The experiment demonstrates relative improvement under compression, not the elimination of information loss.
Training data changes what is worth retaining
Figure 5 compares KVPᴿ, trained on RULER, with KVPˢ, trained on OASST2, on English and Chinese LongBench passage retrieval. KVPᴿ is particularly effective at small budgets. In the English plot at around 1,000 cached tokens, its accuracy is in the 90% range, while KVPˢ is around 40%; these are approximate chart readings.
Conversational training is better suited to the paper’s BoolQ and GovReport evaluations. Retrieval assistance, extended dialogue and summarisation may need different retention priorities. Before adoption, evaluate whether the policy preserves the clues required by the target workload, together with quality, memory use and latency.
What the objective-function ablation shows
Figure 6 compares regression, pairwise ranking, listwise ranking and reinforcement learning. Its vertical axis is not accuracy: it is the negative reward contribution, a normalised training cost for the token at each rank, where lower is better. Focusing only on a few high-attention tokens can leave the remaining ordering poorly learned.
The reward spanning different budgets helps address that problem. Unweighted SoftSort is also competitive outside the smallest capacities. This ablation concerns OASST2, layer 10, KV head 0; it does not establish that reinforcement learning is the uniquely correct objective for every model and head.
Selection overhead is not an end-to-end speed-up
Figure 7 measures importance scoring and selection-index computation for individual KV caches on an NVIDIA B200, with full-model prefill and decoding times as reference lines. It excludes the attention-score recomputation cost for TOVA and SnapKV. At 10,000 tokens, the reported KVP selection time is about 0.71 ms and full-model prefill about 404 ms.
These timings cover different operations. Their ratio must not be advertised as a several-hundred-fold acceleration of the whole service. The parameter-based estimate including all 112 policies increases computation from about 14 to 14.15 GFLOPs per token, roughly 1%. This is a theoretical operation-count increment, not a measured 1% latency or memory penalty, and the paper does not provide a fully optimised end-to-end serving comparison.
Preparation costs and remaining limitations
The OASST2 trace example uses 4,649 conversations on seven nodes with eight GPUs each, producing about 1.2 TB of stored traces. Training one policy on a node with eight H100 GPUs takes under 30 minutes in the reported setup. That figure is not the complete collection-and-training time for all 112 policies.
Future attention remains a proxy; separately learned layers and heads do not fully model their interactions; and the training distribution influences the learned priorities. Evaluation mainly concerns compression around prefill. Reliable eviction throughout very long, continuous generation is not established by these results.
A patent practitioner’s view of the technical contribution
The technical problem is to fit a growing KV cache into limited memory while preserving information needed for subsequent inference. The relevant combination is a per-layer, per-head policy using K, V and position; a ranking shared across budgets; a reward integrating eviction losses; and offline training on collected traces. These components explain how the proposed effect is achieved.
A particularly useful distinction is between future information used to construct training targets and current cache features used at deployment. The results demonstrate quality improvements at given cache budgets, transfer to longer contexts and limited selection overhead under specified conditions. They do not, on their own, establish patentability, a granted patent or the scope of legal protection. KVP’s promise and its dependence on data, objectives and evaluation conditions should be assessed together.
Paper publication record and full text
Figures: CC BY 4.0; original content preserved, converted for web display
Explore our AI patent practice