# Gated DeltaNet-2: separating erase and write

How independent erase and write gates refine fixed-size recurrent memory, with a close reading of the paper’s architecture, benchmarks and limits.

Source: https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write

HOME / NEWS & INSIGHTS NEWS & INSIGHTS Gated DeltaNet-2: separating erase and write How independent erase and write gates refine fixed-size recurrent memory, with a close reading of the paper’s architecture, benchmarks and limits. AI 2026.10.20 published IPLEX 13 min read The paper and its question Ali Hatamizadeh, Yejin Choi and Jan Kautz propose Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention. This analysis concerns arXiv:2605.22791v1, submitted on 21 May 2026; the PDF bears 22 May. It is a preprint, not a claim of journal publication. The paper is available under CC BY 4.0; the reproduced architecture, tables and throughput plot are credited to its authors and NVIDIA. The central question is how to revise a small, compressed memory without disturbing other associations. The advance is a more selective update, rather than simply allocating a larger memory. The following distinguishes the paper’s reported results from our technical and patent-practice interpretation. From a growing cache to a fixed state Softmax attention can consult earlier tokens explicitly, but its KV cache grows with the sequence. Recurrent linear attention instead accumulates key–value associations in a fixed-size matrix. A key addresses an association, a value carries its content and a query reads the current state. Compression reduces storage growth but creates interference when many associations share limited space. DeltaNet updates the difference between an incoming value and the state’s current read. Gated DeltaNet adds learned forgetting; Kimi Delta Attention (KDA) makes forgetting channel-specific. Yet a single scalar beta still ties how strongly old content is erased to how strongly new content is written. Independent decisions on two axes Gated DeltaNet-2 retains channel-wise decay alpha but introduces separate input-dependent erase and write gates through independent linear projections and sigmoid functions. The erase gate b acts on key channels; the write gate w acts on value channels. They form e = b ⊙ k and z = w ⊙ v. After decay gives Sbar = D Sprevious, the old read is r = Sbarᵀe and the update is S = Sbar + k(z − r)ᵀ. The query reads the updated state. Tying both gates to the same scalar beta recovers KDA; also making decay scalar recovers Gated DeltaNet. The relationship between the axes and update terms, rather than the mere presence of two gates, is the technical distinction. Figure 1: Gated DeltaNet-2 hybrid architecture and block design Token processing and parallel training The query and key paths use a linear projection, short causal convolution, SiLU and L2 normalization. The value path uses projection, convolution and SiLU. After the recurrent update, RMS normalization, a separate output gate and projection produce the output. That output gate is distinct from the write gate. Training uses a chunkwise WY formulation, with 64-token chunks. Cumulative channel-wise decay is absorbed into asymmetric erase factors, leaving triangular solves and dense matrix products within each chunk. The backward pass must accumulate gradients using the gated keys and values: channel-wise factors cannot simply be pulled out as a scalar beta could. FP32 decay calculations and state/solve precision matter to the implementation. What was compared The main experiments used 1.3B-parameter models trained on 100B FineWeb-Edu tokens with a 4K training length. Comparators included Mamba-2, Gated DeltaNet, KDA and Mamba-3 SISO/MIMO. Recurrent and hybrid families were evaluated separately; the hybrid adds sliding-window attention with a 2K window. With 16 heads and 128-by-128 state dimensions, the recurrent state contains 262,144 values per layer and batch item. This is not the total memory footprint of the whole model. The hybrid also retains a bounded local attention cache, combining compressed distant context with more exact local interactions. Language and reasoning results Table 2 averages LAMBADA accuracy and the listed commonsense-reasoning tasks. The recurrent model scored 53.11, against KDA’s 52.28 and Gated DeltaNet’s 52.07. The hybrid scored 53.97, against 52.68 and 52.25 respectively. The gains over KDA are 0.83 and 1.29 percentage points, under the reported setup. An improved average does not mean winning every task. Recurrent LAMBADA accuracy was 48.09 versus 49.62 for Gated DeltaNet. Hybrid WikiText perplexity was 15.62, while Mamba-3 SISO achieved 15.54, where lower is better. Table 2: language modeling and commonsense reasoning results Retrieval: gains with substantial remaining errors On RULER’s 4K MK-NIAH-1 multi-key task, recurrent Gated DeltaNet-2 reached 37.8%, versus 28.0% for KDA and 27.8% for Gated DeltaNet. The hybrid reached 48.0% versus KDA’s 40.4%. The recurrent gain of 9.8 percentage points is useful evidence of reduced interference, but 37.8% is far from reliable retrieval. Its recurrent S-NIAH-3 accuracy fell from 89.8% at 2K to 31.8% at 4K; Gated DeltaNet achieved 60.6% at 4K. Table 4 covers SWDE HTML extraction, FDA PDF key–value extraction, SQuAD, TriviaQA, Natural Questions and DROP, with inputs truncated to 2K tokens. Recurrent average accuracy was 29.88 versus KDA’s 28.67; hybrid averages were 42.28 and 40.14. On DROP, the recurrent model trailed KDA, 17.87 versus 21.80. These are not demonstrations of accurate retrieval over hundred-thousand-token documents. Table 3: RULER single-needle and multi-key retrieval accuracy Table 4: real-world retrieval with inputs truncated to 2K tokens Ablations and training throughput The ablations retained the projection weights but averaged one gate across channels at runtime, so parameter count stayed fixed. At 4K MK-NIAH-1, channel-wise write alone scored 30.6, channel-wise erase alone 35.2, and both 37.8. Expanding erase from [0,1] to [0,2] produced 37.6, without a consistent additional benefit. The results support a contribution from both channel structures, with the erase side recovering more of the gain. Figure 2 measures training throughput of hybrid 1.3B models on one H100 at a fixed token budget. Gated DeltaNet-2 moved from 38.0 thousand tokens/s at 2K×8 to 36.1 at 16K×1. KDA remained slightly faster. This is a training measurement with a modest extra computation cost, not a general claim about serving latency or deployment savings. Table 5: gate-channel and erase-range ablations Figure 2: training throughput measured on one H100 Where the evidence stops The architecture may inform systems that accumulate long histories in bounded state or combine local and distant context. That is an inference from its design. The paper does not validate Korean patent analysis, legal-document retrieval or a particular industrial service. The principal evidence is concentrated at 1.3B parameters and 100B training tokens. Larger models, other languages and longer real-world contexts require separate evaluation. The tables do not provide repeated-training variability or confidence intervals. A fixed state still has finite information capacity; finer editing does not turn it into lossless storage. A patent-practice reading of the mechanism The concrete problem is interference when many associations share compressed recurrent memory, compounded by tying key-side erasure and value-side writing to one scalar. The solution joins channel-wise decay with independently gated erase and write in the same delta update. Its distinction from KDA also extends to the corresponding asymmetric parallel formulation and gate-aware gradient calculation. For technical documentation, explain the axes, intermediate quantities and their role in the state update together with the training computation. Effects should be tied to the matched-state benchmarks, ablations and H100 conditions, including task-specific weaknesses and overhead. This is a technical interpretation, not a conclusion that the paper establishes patentability. Takeaway Gated DeltaNet-2 shows how a fixed memory can be edited more selectively: erase and write are learned independently and carried through both recurrence and parallel training. The useful lesson is the connection between the interference problem, the coordinated update and the measured effects. Reading those together with the limits is more informative than treating one best score as proof of universal superiority. The original figure labels and forms are preserved; the text explains the source material. Related service: AI patents Gated DeltaNet-2 paper, version 1 · CC BY 4.0 Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE The paper and its question From a growing cache to a fixed state Independent decisions on two axes Token processing and parallel training What was compared Language and reasoning results Retrieval: gains with substantial remaining errors Ablations and training throughput Where the evidence stops A patent-practice reading of the mechanism Takeaway TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer Duplicate messages and inventive step ↗ Older Trademark registration in Egypt: procedure and practical considerations ↗ Related insights Artificial Intelligence 2026.10.29 SwiLA: organizing memory around context How Switching Linear Attention combines linear regressors within a fixed recurrent state, what its recall experiments establish, and where capacity and computational limits remain. ↗ Read article Artificial Intelligence 2026.10.28 Microsoft’s speech synthesis patent: combining sounds, words and sentence structure US12094447B2 explains how phoneme or character information can be combined with word embeddings and grammatical structure to generate speech. ↗ Read article Artificial Intelligence 2026.10.27 GBLA: reading long recommendation histories efficiently A technical and patent-practice reading of gated bidirectional linear attention, its hybrid encoder and the limits of the reported speed and retrieval results. ↗ Read article

- https://www.iplexlaw.co.kr/en
- https://www.iplexlaw.co.kr/en/blog/category/ai
- https://www.iplexlaw.co.kr/en/ai-patents
- https://arxiv.org/abs/2605.22791v1
- https://creativecommons.org/licenses/by/4.0/
- https://www.iplexlaw.co.kr/ko/blog/gated-deltanet-2-erase-write
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-0
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-1
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-2
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-3
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-4
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-5
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-6
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-7
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-8
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-9
- https://www.iplexlaw.co.kr/en/blog/gated-deltanet-2-erase-write#article-1858-en-10
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog/brunch-case-1859
- https://www.iplexlaw.co.kr/en/blog/egypt-trademark-filing-guide
- https://www.iplexlaw.co.kr/en/blog/swila-switching-linear-attention
- https://www.iplexlaw.co.kr/en/blog/swila-switching-linear-attention
- https://www.iplexlaw.co.kr/en/blog/microsoft-multilevel-tts
- https://www.iplexlaw.co.kr/en/blog/microsoft-multilevel-tts
- https://www.iplexlaw.co.kr/en/blog/gbla-generative-retrieval
- https://www.iplexlaw.co.kr/en/blog/gbla-generative-retrieval