A technical paper review by Yongduck Kim, Korean patent attorney at IPLEX IP LAW FIRM.
The same word can refer to different things in different contexts. An AI system that compresses a conversation into memory needs to preserve those distinctions. If several contexts share a single memory representation, their associations can interfere when the system tries to retrieve information.
Switching Linear Attention, or SwiLA, distributes a fixed-size memory across several linear regression components and adjusts their updates according to the relationship between the incoming input and the existing state. Its central idea is to make retrieval more expressive without allowing memory storage to grow indefinitely.
1. The paper and the version reviewed
Switching Linear Attention is by Hyun Dong Lee, Xavier Gonzalez, Nicolas Zucchet, E. Kelly Buchanan, Emily B. Fox and Scott W. Linderman. The paper lists Stanford University as their affiliation. A footnote states that Xavier Gonzalez was at Stanford during the research and is now at Unconventional AI.
This review concerns arXiv:2609.39034v1, released on 30 September 2026, with DOI 10.48550/arXiv.2609.39034. The original PDF identifies it as a COLM 2026 conference paper. The results and illustrations discussed here refer to that first version.
The paper is available under CC BY 4.0. The reproduced figures and tables are credited to Lee et al., Switching Linear Attention, 2026, with their original identifiers in the captions. The author of this commentary is distinct from the paper’s research team.
2. Why the retrieval function matters: a full record or a compressed state
Conventional softmax attention retains earlier tokens’ keys and values in a KV cache. It compares a query with those keys and combines the relevant values. This provides flexible retrieval, but the cache grows as the context becomes longer.
Linear attention instead accumulates past information in a state matrix of fixed size. It reads from a compact ledger rather than reopening the entire history. A single linear mapping from queries to values, however, has limited ability to represent associations that change with context.
For example, the name Harry calls for different associations in a paragraph about a musician and one about a fictional wizard. SwiLA uses a mixture of regression models to handle cases in which the same cue points to different values depending on the surrounding context.
3. The central idea: memory as a mixture of regressors
The researchers describe memorization as learning a regression function from keys to values, and retrieval as evaluating that function at a query. This is test-time regression, or TTR. The state that changes as tokens arrive consists of fast weights learned within the context; the entire language model is not retrained for each token.
DeltaNet’s basic retrieval operation multiplies a query q by a single state matrix W. SwiLA divides the state into J linear regression components and assigns mixture weights separately for each output dimension. It resembles keeping several specialized ledgers and adjusting their contributions to the answer.
The output sums the components’ predictions, weighted by the query-side mixture probabilities. Switching therefore does not necessarily mean hard routing to one expert. Because those weights depend on the input, the overall retrieval function is more expressive than a single linear transformation.
With D output dimensions, there are J^D possible patterns of component selection, while the number of stored weights scales as J×D². The number of possible combinations must not be confused with actual memory capacity. SwiLA does not claim to preserve the entire past without loss in a fixed-size state.
4. Processing one token: prediction error determines the update
First, the input token produces queries, keys, values and mixture probabilities. Second, each regression component uses its existing state to predict a value for the current key. The difference between that prediction and the actual value is its prediction error.
Third, the model combines the input-derived prior probabilities with the components’ prediction errors to calculate their responsibilities. The same input can receive different responsibilities depending on what is already stored. This distinguishes the mechanism from a router that allocates memory using only the input.
Fourth, each component’s fast weights receive an update proportional to learning rate × responsibility × prediction error × key. Fifth, the query is applied to the updated components, and their outputs are combined with the query-side mixture weights. The researchers derive these steps from online expectation–maximization for a mixture of linear regressions.
EM provides a way to estimate the probabilities of latent component assignments and update the model. Here the recurrence uses a single stochastic-gradient update of the fast weights per token. It does not run a lengthy optimization loop to convergence whenever a token arrives.
5. Preserving context and forgetting: Temporal SwiLA and Gated SwiLA
Temporal SwiLA accounts for context persisting across consecutive tokens. It blends the previous probability distribution over components with a distribution derived from the current input, then carries the result forward. This tractable approximation to hidden Markov model filtering allows component preferences to persist even when contextual cues are not repeated within a paragraph.
Gated SwiLA adds an input-dependent forgetting gate that attenuates older state. The paper distinguishes applying decay before and after responsibility calculation. Its implemented Gated SwiLA uses the formulation in which prediction errors and responsibilities are computed from the decayed state.
Multiple components are of little use if updates concentrate in only a few of them. The researchers introduce an auxiliary loss encouraging balanced use across components alongside specialization for individual tokens. They also describe input-dependent noise, but disable router and temporal-gate noise in the main 374M language-model experiments. Not every proposed technique is active in every experiment.
6. Synthetic experiments: one key, different values in different contexts
The softmax approximation experiment uses synthetic sequences of length 512 and dimension 32, with 65,536 training sequences and 2,048 test sequences. SwiLA and DeltaNet are compared at equal recurrent-state budgets. The best SwiLA configuration at each budget achieves a higher test R². This is a result for the specified synthetic inputs and configuration search, not a guarantee of equally accurate approximation for every input.
The context-dependent associative recall task, CDAR, supplies 32 key–value pairs in each of four contexts, for 128 associations in total. The same key maps to a different value in each context, and context tokens are not continually repeated. The model must retain both the associations and the active context. Sequences are 1,024 tokens long.
In Figure 2A, Temporal SwiLA with one head, dimension 32 and four mixture components is the smallest configuration reaching near-perfect accuracy, using a state size of 4,096. The plot reports the best accuracy found across learning rates and random seeds. It is not a measure of average training stability or a real-world service success rate.
RegBench presents example strings generated by random probabilistic finite automata and tests whether a model learns their next-token distributions within the context. The comparison uses eight-layer models and matches each recurrent layer’s state size at 16,384. Temporal SwiLA and DN-MoM perform strongly when training data are limited, but the Transformer reaches the highest accuracy as more data become available. SwiLA does not lead at every data scale.
7. Language-model results under matched model and training conditions
The main comparison trains models with approximately 374 million parameters, 24 layers and hidden dimension 1,024 on about 15 billion FineWeb-Edu tokens. Training sequences contain 4,096 tokens, and training uses eight H100 GPUs. Feed-forward intermediate dimensions are adjusted slightly to match total parameter counts across architectures.
The SwiLA variants use eight heads, four mixture components per head, and 64-dimensional keys and values, giving 131,072 main recurrent-state scalars per layer. DeltaNet, GDN and KDA are matched to that main-state budget. Matching this quantity does not mean that total runtime memory or computational work is identical.
Average commonsense reasoning accuracy is 45.3% for Transformer++, 43.7% for DeltaNet, 45.0% for Gated DeltaNet and 46.6% for KDA. SwiLA scores 44.7%; Gated SwiLA and Gated Temporal SwiLA each score 45.4%. The gated variants exceed Transformer++ by 0.1 percentage points, while KDA scores higher. The paper does not establish that this small difference is statistically conclusive or that it generalizes to much larger models.
Average recall-task scores are 17.3% for DeltaNet, 19.2% for Gated DeltaNet, 20.3% for KDA and 20.6% for Gated Temporal SwiLA. Transformer++ reaches 28.3% and FoX 33.7%, leaving a substantial gap. Improving recall within a fixed state is different from fully replacing a model that retains its complete KV cache.
Hybrid Temporal SwiLA interleaves three recurrent layers with one softmax-attention layer and scores 31.8%, above Hybrid GDN’s 29.5% and Transformer++’s 28.3%. Its full-attention layers still retain KV caches. The fixed-state property of pure SwiLA therefore does not apply to the hybrid model as a whole.
8. When throughput improves: training and generation are separate measurements
Unlike the accuracy experiments, the throughput study measures 1.3-billion-parameter models on one H100 GPU. For training sequences of 2K, 4K, 8K and 16K tokens, batch sizes are 64, 32, 16 and 8 respectively, keeping each update at 131,072 tokens. Timings include the forward and backward passes, gradient clipping and the AdamW update.
For inference, each synthetic prompt contains 2,048 tokens and is extended by 128 tokens using greedy decoding. Batch size increases from 1 to 128. The measurement covers the entire generation pass, including both prefill and decoding. Configurations at and beyond an out-of-memory failure are omitted from completed measurements.
The researchers report overall inference throughput four to five times that of the Transformer baseline. This includes the benefit of supporting larger batches without a KV cache. It does not mean every batch runs four to five times faster or that one user’s response arrives that much sooner. Against GDN, SwiLA trains about twice as slowly, runs inference about 1.25–1.5 times more slowly, and uses more memory.
9. Remaining limitations and practical conditions
First, memory capacity is still fixed. A longer context does not create more storage, so interference and capacity limits remain. The paper explicitly truncates recall-test inputs to 2K tokens; those results alone do not establish accuracy on extremely long documents.
Second, responsibilities that depend on the existing state produce a nonlinear recurrence that is difficult to parallelize during training. The experiments use a fast sequential Triton kernel. The appendix proposes Newton iterations and chunk-based parallelization, but those proposals must not be described as having already solved the training-speed limitation in the main experiments.
Third, more experts do not automatically improve performance. In a separate ablation using 115M-parameter models and five billion training tokens, increasing components from four to sixteen under the same state budget reduces each component’s projection dimensions and slightly worsens performance. Expert count must be balanced against each component’s representational capacity.
Fourth, basic SwiLA is vulnerable to expert collapse. In the ablation, adding the auxiliary loss reduces test perplexity from 36.769 to 16.807. For Temporal SwiLA, the corresponding values are 16.718 and 16.715, a much smaller difference. These results belong to the ablation and must not be mixed with the main 374M comparison.
10. Technical distinctions from a patent attorney’s perspective
The technical problem is to preserve and retrieve context-dependent key–value associations more accurately within a limited recurrent state. The proposed solution combines linear regression mixtures, weights specific to each output dimension, responsibilities reflecting the existing state’s prediction errors, and state updates governed by those responsibilities.
Compared with input-only memory selection, SwiLA considers how well the current memory explains the incoming input. The Temporal extension adds persistence of context, while the Gated extension adds state decay. Separating these roles and explaining how they interact is essential to an accurate account of the architecture.
The demonstrated effects include better synthetic-task results at specified state budgets and improved recall in the 374M experiments. They coexist with higher training and inference costs and a remaining recall gap relative to full attention. In my view, the contribution lies less in partitioning memory alone than in designing the update rule and retrieval function together within a consistent probabilistic model. This is technical commentary, not a determination of patentability or infringement.
11. Closing observations
SwiLA retains linear attention’s fixed-size memory while making retrieval more expressive through context-dependent mixtures of regression components. Two central features are the effect of existing-state prediction error on subsequent updates and an explicit mechanism for context persistence.
The experiments show that recall can improve within a fixed state, while the capacity difference from full-history attention and the cost of sequential training remain. Assessing suitability for an application requires considering context length, recall accuracy, batch throughput and training cost together.
The original figure labels and forms are preserved; the text explains the source material.
Explore IPLEX’s AI technology and patent analysis
Research paper: Switching Linear Attention