The paper and its question
Ali Hatamizadeh, Yejin Choi and Jan Kautz propose Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. This analysis concerns arXiv:2605.22791v1, submitted on 21 May 2026; the PDF bears 22 May. It is a preprint, not a claim of journal publication. The paper is available under CC BY 4.0; the reproduced architecture, tables and throughput plot are credited to its authors and NVIDIA.
The central question is how to revise a small, compressed memory without disturbing other associations. The advance is a more selective update, rather than simply allocating a larger memory. The following distinguishes the paper’s reported results from our technical and patent-practice interpretation.
From a growing cache to a fixed state
Softmax attention can consult earlier tokens explicitly, but its KV cache grows with the sequence. Recurrent linear attention instead accumulates key–value associations in a fixed-size matrix. A key addresses an association, a value carries its content and a query reads the current state. Compression reduces storage growth but creates interference when many associations share limited space.
DeltaNet updates the difference between an incoming value and the state’s current read. Gated DeltaNet adds learned forgetting; Kimi Delta Attention (KDA) makes forgetting channel-specific. Yet a single scalar beta still ties how strongly old content is erased to how strongly new content is written.
Independent decisions on two axes
Gated DeltaNet-2 retains channel-wise decay alpha but introduces separate input-dependent erase and write gates through independent linear projections and sigmoid functions. The erase gate b acts on key channels; the write gate w acts on value channels. They form e = b ⊙ k and z = w ⊙ v.
After decay gives Sbar = D Sprevious, the old read is r = Sbarᵀe and the update is S = Sbar + k(z − r)ᵀ. The query reads the updated state. Tying both gates to the same scalar beta recovers KDA; also making decay scalar recovers Gated DeltaNet. The relationship between the axes and update terms, rather than the mere presence of two gates, is the technical distinction.
Token processing and parallel training
The query and key paths use a linear projection, short causal convolution, SiLU and L2 normalization. The value path uses projection, convolution and SiLU. After the recurrent update, RMS normalization, a separate output gate and projection produce the output. That output gate is distinct from the write gate.
Training uses a chunkwise WY formulation, with 64-token chunks. Cumulative channel-wise decay is absorbed into asymmetric erase factors, leaving triangular solves and dense matrix products within each chunk. The backward pass must accumulate gradients using the gated keys and values: channel-wise factors cannot simply be pulled out as a scalar beta could. FP32 decay calculations and state/solve precision matter to the implementation.
What was compared
The main experiments used 1.3B-parameter models trained on 100B FineWeb-Edu tokens with a 4K training length. Comparators included Mamba-2, Gated DeltaNet, KDA and Mamba-3 SISO/MIMO. Recurrent and hybrid families were evaluated separately; the hybrid adds sliding-window attention with a 2K window.
With 16 heads and 128-by-128 state dimensions, the recurrent state contains 262,144 values per layer and batch item. This is not the total memory footprint of the whole model. The hybrid also retains a bounded local attention cache, combining compressed distant context with more exact local interactions.
Language and reasoning results
Table 2 averages LAMBADA accuracy and the listed commonsense-reasoning tasks. The recurrent model scored 53.11, against KDA’s 52.28 and Gated DeltaNet’s 52.07. The hybrid scored 53.97, against 52.68 and 52.25 respectively. The gains over KDA are 0.83 and 1.29 percentage points, under the reported setup.
An improved average does not mean winning every task. Recurrent LAMBADA accuracy was 48.09 versus 49.62 for Gated DeltaNet. Hybrid WikiText perplexity was 15.62, while Mamba-3 SISO achieved 15.54, where lower is better.
Retrieval: gains with substantial remaining errors
On RULER’s 4K MK-NIAH-1 multi-key task, recurrent Gated DeltaNet-2 reached 37.8%, versus 28.0% for KDA and 27.8% for Gated DeltaNet. The hybrid reached 48.0% versus KDA’s 40.4%. The recurrent gain of 9.8 percentage points is useful evidence of reduced interference, but 37.8% is far from reliable retrieval. Its recurrent S-NIAH-3 accuracy fell from 89.8% at 2K to 31.8% at 4K; Gated DeltaNet achieved 60.6% at 4K.
Table 4 covers SWDE HTML extraction, FDA PDF key–value extraction, SQuAD, TriviaQA, Natural Questions and DROP, with inputs truncated to 2K tokens. Recurrent average accuracy was 29.88 versus KDA’s 28.67; hybrid averages were 42.28 and 40.14. On DROP, the recurrent model trailed KDA, 17.87 versus 21.80. These are not demonstrations of accurate retrieval over hundred-thousand-token documents.
Ablations and training throughput
The ablations retained the projection weights but averaged one gate across channels at runtime, so parameter count stayed fixed. At 4K MK-NIAH-1, channel-wise write alone scored 30.6, channel-wise erase alone 35.2, and both 37.8. Expanding erase from [0,1] to [0,2] produced 37.6, without a consistent additional benefit. The results support a contribution from both channel structures, with the erase side recovering more of the gain.
Figure 2 measures training throughput of hybrid 1.3B models on one H100 at a fixed token budget. Gated DeltaNet-2 moved from 38.0 thousand tokens/s at 2K×8 to 36.1 at 16K×1. KDA remained slightly faster. This is a training measurement with a modest extra computation cost, not a general claim about serving latency or deployment savings.
Where the evidence stops
The architecture may inform systems that accumulate long histories in bounded state or combine local and distant context. That is an inference from its design. The paper does not validate Korean patent analysis, legal-document retrieval or a particular industrial service.
The principal evidence is concentrated at 1.3B parameters and 100B training tokens. Larger models, other languages and longer real-world contexts require separate evaluation. The tables do not provide repeated-training variability or confidence intervals. A fixed state still has finite information capacity; finer editing does not turn it into lossless storage.
A patent-practice reading of the mechanism
The concrete problem is interference when many associations share compressed recurrent memory, compounded by tying key-side erasure and value-side writing to one scalar. The solution joins channel-wise decay with independently gated erase and write in the same delta update. Its distinction from KDA also extends to the corresponding asymmetric parallel formulation and gate-aware gradient calculation.
For technical documentation, explain the axes, intermediate quantities and their role in the state update together with the training computation. Effects should be tied to the matched-state benchmarks, ablations and H100 conditions, including task-specific weaknesses and overhead. This is a technical interpretation, not a conclusion that the paper establishes patentability.
Takeaway
Gated DeltaNet-2 shows how a fixed memory can be edited more selectively: erase and write are learned independently and carried through both recurrence and parallel training.
The useful lesson is the connection between the interference problem, the coordinated update and the measured effects. Reading those together with the limits is more informative than treating one best score as proof of universal superiority.
The original figure labels and forms are preserved; the text explains the source material.
Gated DeltaNet-2 paper, version 1 · CC BY 4.0