Paper and scope of this review
Test-Time Training Done Right introduces Large Chunk Test-Time Training (LaCT). Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T. Freeman and Hao Tan list MIT and Adobe Research as their affiliations. The paper first appeared on 29 May 2025 as arXiv:2505.23884, DOI 10.48550/arXiv.2505.23884. Its ICLR 2026 publication is confirmed separately; this review consistently uses the 32-page arXiv v1 for mechanisms, figures and numbers. Figures are attributed to Zhang et al. under CC BY 4.0.
Fast weights are temporary context memory
Ordinary attention reads stored keys and values through queries. Test-time training instead stores the key–value relationship in adaptable weights W of a small network and reads it with a query. These fast weights change for the current input; the base model’s slow weights stay fixed at inference. This is not supervised retraining of the entire model. Updating every token or every 16–64 tokens can repeatedly move weights through memory while leaving GPU arithmetic units underused. LaCT groups 2K to roughly 1M tokens per update across different tasks. Figure 1 presents distinct throughput, state-size, loss and view-synthesis measurements, not one universal service-speed multiplier.
Local attention and nonlinear memory work together
Window attention captures local order or spatial relations. The large-chunk update treats tokens as a set, making that local component important. LaCT’s default nonlinear memory is a three-matrix SwiGLU MLP. A negative inner-product loss aligns network outputs for keys with values; gradients over the chunk incorporate token-specific learning rates. Queries then read the fast weights. Fixed state size does not mean unlimited capacity. The paper compares gradient descent, momentum and Muon; Muon transforms matrix gradients using Newton–Schulz iterations, and updated fast weights are L2-normalised. Fewer, larger updates make richer states and update rules more practical.
Update order must respect the task
Inputs produce queries, keys, values and learning rates; local attention and memory outputs are combined before feed-forward processing. Language and video variants can share Q, K and V between attention and TTT and add their outputs. In language modelling, the current chunk reads memory containing only earlier chunks, then updates it for the next chunk. Causal sliding-window attention supplies within-chunk dependencies. Otherwise future tokens could leak into predictions. With posed image sets, memory can first absorb the inputs and then answer target-view queries. Video memory updates use clean frame chunks; noisy chunks read previously accumulated information.
View synthesis: the measured setting matters
The object models train on Objaverse and are evaluated on Google Scanned Objects; scene experiments use DL3DV. PSNR measures reconstruction quality, with higher values better here. Table 2 uses an A100, 48 images at 512×512, about 196K input tokens and 4K decoding tokens. Full attention reports 16.1 seconds for input processing and 2.3 FPS rendering; Perceiver reports 16.8 seconds and 34.4 FPS; LaCT reports 1.4 seconds and 38.7 FPS. Model sizes are 284M, 287M and 312M. The approximately 11.5-fold input-latency ratio is specific to this setting, not an LLM response-speed claim. Quality is close to full attention, without winning at every point. The largest scene setting uses 128 images at 960×536, about one million visual tokens, not a million-token language evaluation.
Language modelling: quality and throughput differ
Training sequences contain 32,768 tokens. The 760M model uses 40B training tokens and a 2,048-token window; the 3B model uses 60B tokens and a 4,096-token window. GLA and DeltaNet baselines receive the same sliding-window attention. Per-position loss and S-NIAH retrieval favour LaCT with Muon over those recurrent baselines, while full attention remains better at some points. Table 3 measures 3B training at 32K on A100-40GB GPUs: Transformer 4.1K tokens/s, window-only 6.4K, GLA 5.0K, DeltaNet 5.1K, LaCT gradient descent 5.0K, momentum 4.9K and Muon 4.3K. Muon’s quality advantage has an update cost; it is not the highest-throughput option.
Large chunks alone do not explain the gains
Figure 8 separates state representation from update granularity. A smaller nonlinear SwiGLU state can outperform a larger linear state. A linear chunk update with equal state size does not beat the GLA baseline; the combination of a large nonlinear state and Muon does. In the largest tested configuration, fast weights amount to about 40% of model parameters, not a constant for every LaCT model. Capacity, memory footprint and update cost must be considered together.
Video and distributed computation
The authors adapt pretrained Wan 2.1 to 1.3B and 14B autoregressive video diffusion models. An 8.8-second clip in the 14B setting contains 56,160 visual tokens. Approximately 107K tokens during training include interleaved clean and noisy chunks and are a different quantity. After 5,000 training iterations, evaluation uses 2,000 videos and five noise timesteps. Denoising loss is not a direct measure of human preference or long-video quality; proprietary training videos also limit reproducibility. Splitting a large chunk across devices and combining update gradients enables context parallelism. Reported overhead of 1–3% is specific to the view-synthesis training setup, not every network or device count.
Limitations and deployment questions
SwiGLU and linear fast-weight components lack the uniform query–key rotation invariance of some attention mechanisms; the practical effect remains underexplored. View synthesis assumes known camera poses. Reconstruction without poses, complex language reasoning and much larger language models were not established by these experiments. Compressing context into a fixed state calls for separate tests of exact-detail retention, request concurrency, generation latency and memory use. Those service-level checks are our implementation considerations, not additional reported results.
Technical contribution from a patent-analysis perspective
The problem is the interaction between memory traffic, frequent updates and limited use of GPU parallelism. The proposed combination joins large-chunk gradient computation, nonlinear fast weights, normalised update rules and window attention, with task-appropriate read/update order and optional context parallelism. Attention plus recurrence was already known: the distinctive technical account lies in how a large parallel update unit enables a richer state and optimiser. Reported effects should retain their conditions—view-synthesis latency and rendering, or language loss and retrieval at 760M/3B scales—alongside the quality–throughput trade-off. This is an analysis of the paper’s mechanisms, not a conclusion that those mechanisms necessarily satisfy patentability requirements.
Explore AI patents and technical analysis