# LaCT: updating an AI model’s temporary memory in large chunks

A technical review of Large Chunk Test-Time Training, its fast weights, GPU efficiency, causal processing and task-specific experimental limits.

Source: https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training

HOME / NEWS & INSIGHTS NEWS & INSIGHTS LaCT: updating an AI model’s temporary memory in large chunks A technical review of Large Chunk Test-Time Training, its fast weights, GPU efficiency, causal processing and task-specific experimental limits. AI 2026.10.13 published IPLEX 13 min read Paper and scope of this review Test-Time Training Done Right introduces Large Chunk Test-Time Training (LaCT). Tianyuan Zhang, Sai Bi, Yicong Hong, Kai Zhang, Fujun Luan, Songlin Yang, Kalyan Sunkavalli, William T. Freeman and Hao Tan list MIT and Adobe Research as their affiliations. The paper first appeared on 29 May 2025 as arXiv:2505.23884, DOI 10.48550/arXiv.2505.23884. Its ICLR 2026 publication is confirmed separately; this review consistently uses the 32-page arXiv v1 for mechanisms, figures and numbers. Figures are attributed to Zhang et al. under CC BY 4.0. Fast weights are temporary context memory Ordinary attention reads stored keys and values through queries. Test-time training instead stores the key–value relationship in adaptable weights W of a small network and reads it with a query. These fast weights change for the current input; the base model’s slow weights stay fixed at inference. This is not supervised retraining of the entire model. Updating every token or every 16–64 tokens can repeatedly move weights through memory while leaving GPU arithmetic units underused. LaCT groups 2K to roughly 1M tokens per update across different tasks. Figure 1 presents distinct throughput, state-size, loss and view-synthesis measurements, not one universal service-speed multiplier. Figure 1. Separate measurements of GPU throughput, state size, loss and training time. Zhang et al., arXiv v1, CC BY 4.0. Local attention and nonlinear memory work together Window attention captures local order or spatial relations. The large-chunk update treats tokens as a set, making that local component important. LaCT’s default nonlinear memory is a three-matrix SwiGLU MLP. A negative inner-product loss aligns network outputs for keys with values; gradients over the chunk incorporate token-specific learning rates. Queries then read the fast weights. Fixed state size does not mean unlimited capacity. The paper compares gradient descent, momentum and Muon; Muon transforms matrix gradients using Newton–Schulz iterations, and updated fast weights are L2-normalised. Fewer, larger updates make richer states and update rules more practical. Figure 2. Window attention, large-chunk TTT and feed-forward processing. Zhang et al., arXiv v1, CC BY 4.0. Update order must respect the task Inputs produce queries, keys, values and learning rates; local attention and memory outputs are combined before feed-forward processing. Language and video variants can share Q, K and V between attention and TTT and add their outputs. In language modelling, the current chunk reads memory containing only earlier chunks, then updates it for the next chunk. Causal sliding-window attention supplies within-chunk dependencies. Otherwise future tokens could leak into predictions. With posed image sets, memory can first absorb the inputs and then answer target-view queries. Video memory updates use clean frame chunks; noisy chunks read previously accumulated information. Figure 3. Update/Apply order and causal information access. Zhang et al., arXiv v1, CC BY 4.0. View synthesis: the measured setting matters The object models train on Objaverse and are evaluated on Google Scanned Objects; scene experiments use DL3DV. PSNR measures reconstruction quality, with higher values better here. Table 2 uses an A100, 48 images at 512×512, about 196K input tokens and 4K decoding tokens. Full attention reports 16.1 seconds for input processing and 2.3 FPS rendering; Perceiver reports 16.8 seconds and 34.4 FPS; LaCT reports 1.4 seconds and 38.7 FPS. Model sizes are 284M, 287M and 312M. The approximately 11.5-fold input-latency ratio is specific to this setting, not an LLM response-speed claim. Quality is close to full attention, without winning at every point. The largest scene setting uses 128 images at 960×536, about one million visual tokens, not a million-token language evaluation. Figure 4. View-synthesis quality, input latency and number of images. Zhang et al., arXiv v1, CC BY 4.0. Language modelling: quality and throughput differ Training sequences contain 32,768 tokens. The 760M model uses 40B training tokens and a 2,048-token window; the 3B model uses 60B tokens and a 4,096-token window. GLA and DeltaNet baselines receive the same sliding-window attention. Per-position loss and S-NIAH retrieval favour LaCT with Muon over those recurrent baselines, while full attention remains better at some points. Table 3 measures 3B training at 32K on A100-40GB GPUs: Transformer 4.1K tokens/s, window-only 6.4K, GLA 5.0K, DeltaNet 5.1K, LaCT gradient descent 5.0K, momentum 4.9K and Muon 4.3K. Muon’s quality advantage has an update cost; it is not the highest-throughput option. Figure 5. Per-position loss and retrieval accuracy for 760M and 3B language models. Zhang et al., arXiv v1, CC BY 4.0. Large chunks alone do not explain the gains Figure 8 separates state representation from update granularity. A smaller nonlinear SwiGLU state can outperform a larger linear state. A linear chunk update with equal state size does not beat the GLA baseline; the combination of a large nonlinear state and Muon does. In the largest tested configuration, fast weights amount to about 40% of model parameters, not a constant for every LaCT model. Capacity, memory footprint and update cost must be considered together. Figure 8. Linear/nonlinear states and token/chunk updates compared. Zhang et al., arXiv v1, CC BY 4.0. Video and distributed computation The authors adapt pretrained Wan 2.1 to 1.3B and 14B autoregressive video diffusion models. An 8.8-second clip in the 14B setting contains 56,160 visual tokens. Approximately 107K tokens during training include interleaved clean and noisy chunks and are a different quantity. After 5,000 training iterations, evaluation uses 2,000 videos and five noise timesteps. Denoising loss is not a direct measure of human preference or long-video quality; proprietary training videos also limit reproducibility. Splitting a large chunk across devices and combining update gradients enables context parallelism. Reported overhead of 1–3% is specific to the view-synthesis training setup, not every network or device count. Limitations and deployment questions SwiGLU and linear fast-weight components lack the uniform query–key rotation invariance of some attention mechanisms; the practical effect remains underexplored. View synthesis assumes known camera poses. Reconstruction without poses, complex language reasoning and much larger language models were not established by these experiments. Compressing context into a fixed state calls for separate tests of exact-detail retention, request concurrency, generation latency and memory use. Those service-level checks are our implementation considerations, not additional reported results. Technical contribution from a patent-analysis perspective The problem is the interaction between memory traffic, frequent updates and limited use of GPU parallelism. The proposed combination joins large-chunk gradient computation, nonlinear fast weights, normalised update rules and window attention, with task-appropriate read/update order and optional context parallelism. Attention plus recurrence was already known: the distinctive technical account lies in how a large parallel update unit enables a richer state and optimiser. Reported effects should retain their conditions—view-synthesis latency and rendering, or language loss and retrieval at 760M/3B scales—alongside the quality–throughput trade-off. This is an analysis of the paper’s mechanisms, not a conclusion that those mechanisms necessarily satisfy patentability requirements. LaCT paper and source figures Figure licence: CC BY 4.0 Explore AI patents and technical analysis Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE Paper and scope of this review Fast weights are temporary context memory Local attention and nonlinear memory work together Update order must respect the task View synthesis: the measured setting matters Language modelling: quality and throughput differ Large chunks alone do not explain the gains Video and distributed computation Limitations and deployment questions Technical contribution from a patent-analysis perspective TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer General powers of attorney and correction of Korean patent formalities ↗ Older Design drawings, adjustable products and the scope of protection ↗ Related insights AI 2026.10.15 PaTH Attention: encoding the path between tokens An analysis of input-dependent Householder transformations, state tracking and the measured benefits and limits of PaTH Attention. ↗ Read article AI 2026.10.14 Huawei Pangu Weather: linking forecast models across time intervals A technical reading of Huawei’s US20250258318A1 publication: selecting forecast intervals, passing predicted states and combining weather data. ↗ Read article AI 2026.10.12 Adobe’s AI image-editing patent: keeping originals and generated alternatives in layers An explanation of US12462456B2: input masks, subject masks and generative layers that preserve alternatives and editing context. ↗ Read article

- https://www.iplexlaw.co.kr/en
- https://www.iplexlaw.co.kr/en/blog/category/ai
- https://arxiv.org/abs/2505.23884v1
- https://creativecommons.org/licenses/by/4.0/
- https://www.iplexlaw.co.kr/en/ai-patents
- https://www.iplexlaw.co.kr/ko/blog/lact-large-chunk-test-time-training
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-0
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-1
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-2
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-3
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-4
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-5
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-6
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-7
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-8
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training#case-1827-en-9
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog/brunch-case-1828
- https://www.iplexlaw.co.kr/en/blog/brunch-case-1826
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations
- https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models
- https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models
- https://www.iplexlaw.co.kr/en/blog/adobe-non-destructive-generative-editing
- https://www.iplexlaw.co.kr/en/blog/adobe-non-destructive-generative-editing