
A long context window does not by itself ensure that an AI system can retain and use information from a long history. Titans tackles the problem by changing a dedicated neural memory while processing new input, alongside attention to the current context.
This analysis follows the first arXiv version of “Titans: Learning to Memorize at Test Time”. It examines what is updated, how memory and attention interact, and what the reported experiments support, with the original diagrams and tables retained.
The paper and the version reviewed
The paper is by Ali Behrouz, Peilin Zhong and Vahab Mirrokni of Google Research. Its identifier is arXiv:2501.00663 and its DOI is 10.48550/arXiv.2501.00663. The reviewed v1 was submitted on 31 December 2024 and contains 27 pages.
A later publication appears in NeurIPS 2025. That later record is distinguished here from the v1 text: architecture descriptions, figures and numerical results below refer to v1, rather than being silently mixed with subsequent versions.
Version 1 is available under CC BY 4.0. The three researchers are the original authors of the paper's diagrams and tables; each reproduced item is identified by its original figure or table number in the caption.
Why a larger window is not the whole answer
Full attention can model direct relationships between tokens, but its computational and memory costs rise rapidly with context length. Recurrent alternatives compress the past into a fixed-size state, improving scalability while risking loss of information. The capacity to accept a long sequence and the ability to use its earlier facts are different measures.
One useful analogy is a desk for the current task and notes for earlier information. In Titans, however, the “notes” are associations encoded in neural-network weights, not a readable transcript or a conventional document archive. Short-term attention and learned long-term memory therefore play distinct roles.
Learning associations at test time
The memory maps a key k to a value v. Equation 12 uses the squared error ‖M(k) − v‖² to train that association. In the inner loop, the memory weights M change; the projections producing keys and values and other architecture parameters are learned through the outer training process. This is not wholesale retraining of an entire large language model for every incoming passage.
The paper uses the gradient of the associative-memory loss as a “surprise” signal. Information that the current memory handles poorly can produce a larger update. Surprise here is an optimisation quantity; it does not certify that a statement is true or important in the user's intended sense.
Momentum carries forward the effect of recent surprise, and an input-dependent forgetting coefficient controls how much of the previous memory is retained. This combines the current gradient, a trace of earlier updates and adaptive decay, rather than accumulating every item indefinitely.
Retrieval is a separate operation: a query is passed forward through the memory without updating its weights. Distinguishing memory writing from memory reading is essential to understanding what continues learning at test time.
MAC: memory supplied as context
Memory as a Context (MAC) divides the input into segments. The current segment queries the existing long-term memory; the retrieved information is combined with persistent-memory tokens and the current segment before attention is applied. Figure 2 shows these interacting branches.

The attention output then updates long-term memory for the next segment and contributes, together with memory output, to the final result. The flow is retrieval, combination, attention and memory update: it is more specific than simply attaching a generic memory to a Transformer.
Persistent memory has a different meaning here. It consists of learned, input-independent parameters representing task knowledge and remains fixed at test time. It should not be described as a facility that permanently saves each user's conversations.
MAG, MAL and the memory-only comparison
Memory as a Gate (MAG) processes input through a neural-memory branch and a sliding-window-attention branch, then combines their outputs through gating. The branches can contribute complementary information, with attention handling the current window and neural memory carrying earlier context. Figure 4 depicts this arrangement.

Memory as a Layer (MAL) instead places memory before attention in a sequence of layers. Information is first processed and compressed by memory, then passed to attention. Figure 5 illustrates this serial arrangement, whose intermediate representation can become a bottleneck and limit the complementary processing available in the other variants.

The paper also evaluates the long-term memory module on its own, called LMM, without attention. That comparison helps separate the contribution of the memory-update mechanism from the benefit of combining it with attention.
Experimental scale and training conditions
The experimental setup lists models with 170M, 340M, 400M and 760M parameters. The first three were trained on 15 billion FineWeb-Edu tokens and the 760M models on 30 billion. Table 1 reports the 340M, 400M and 760M settings. Parameter count and token budget must therefore accompany comparisons.
The training length was 4,096 tokens, with the Llama 2 tokenizer and a 32K vocabulary. Training used AdamW, an initial learning rate of 0.0004 with cosine scheduling, batches of 500,000 tokens and weight decay of 0.1. Baselines included Transformer++, the Mamba family, DeltaNet, TTT and attention/recurrent hybrids.
Reading the language-model results carefully
At 760M parameters and 30B tokens, Table 1 reports average task scores of 48.69 for Transformer++, 49.69 for Gated DeltaNet, 51.08 for Samba and 51.49 for Gated DeltaNet-H2. LMM scored 51.56, MAC 52.51, MAG 52.50 and MAL 50.97. These numbers do not support saying that every Titans variant beats every baseline.
The Wiki perplexities were 19.93 for MAC, 18.61 for MAG and 19.07 for MAL, compared with 19.88 for Gated DeltaNet-H2; lower perplexity is better. MAC's slightly higher average task score does not make it the best model on every language-modelling measure.
At 400M parameters and 15B tokens, MAC and MAG had average scores of 48.65 and 48.60. These are scores for the listed benchmarks and training conditions, not a universal measure of intelligence or proof of superiority over commercial large language models.
Retrieval over long contexts
The RULER single-needle tests use S-NIAH-PK, S-NIAH-N and S-NIAH-W at 2K, 4K, 8K and 16K sequence lengths. They test whether relevant information can be recovered from distracting text, rather than merely whether the model accepts the input. Table 2 retains the original results.

At 16K, MAC achieved 98.4, 97.4 and 95.2 across the three tasks; LMM scored 96.2, 80.2 and 80.6, while TTT scored 88.4, 4.4 and 0.0. MAG reached 98.6 on S-NIAH-N, higher than MAC's 97.4, so MAC was not the winner in every cell.
BABILong requires reasoning across facts scattered through much longer documents, and the paper reports operation beyond two million tokens. Its few-shot and fine-tuned settings, model sizes and RAG baselines differ. The reported comparisons, including those involving GPT-4, should not be recast as uniform conditions or guaranteed success on all business documents.
Memory depth, speed and practical limits
The study compares memory modules with one to four MLP layers and linear memory. Deeper memory improved perplexity in the tested settings but reduced throughput. These depth experiments used the memory module and a subset of the Pile dataset; their results should not be treated as a complete cost estimate for every Titans architecture.
The implementation groups gradients into chunks, uses matrix multiplications and parallel processing of momentum. This does not mean every nonlinear update is an unrestricted single parallel operation. The paper also notes that its memory module was somewhat slower than Mamba2 and Gated DeltaNet.
Compressed associations are not a verbatim, provenance-preserving database. Updating or forgetting weights does not by itself establish factual accuracy, exact citation recovery or deletion guarantees. The v1 experiments, whose largest listed model is 760M parameters, should be interpreted within that scope.
What the ablations tell a technical or patent reviewer
For a technical assessment, the useful question is how key–value memory, momentum, adaptive forgetting and attention are connected. “The AI remembers” is too broad to identify the mechanism or the relationship among its components.

Table 5 gives LMM a long-context score of 92.68. Removing momentum lowers it to 87.12, removing weight decay to 85.60, and using linear memory to 85.34. These ablations help identify the contribution of the components. The table's language-modelling perplexity must not automatically be labelled as Table 1's Wiki perplexity.
The long-context scores in that table are 97.95 for MAC, 96.70 for MAG and 96.91 for MAL. Such results inform comparison of architecture choices; they do not themselves establish the validity of a patent claim or prove that a commercial product implements the same system.
The architectural lesson
Titans links three decisions: how to retain earlier information, how to attend to current input, and how to update or forget memory as processing continues. Its contribution is the concrete interaction of these mechanisms rather than the label “long-term memory” alone.
A useful memory is not necessarily an exact copy of the original material. Assessments should keep the tested scale, evaluation conditions and computational trade-offs visible, while distinguishing the proposed neural mechanism from promises about production-system behaviour.
The original figure labels and forms are preserved; the text explains the source material.
The numerical results and architecture discussed here are those of the 27-page arXiv v1. They should not be treated as interchangeable with a later conference version or as proof of a commercial product’s performance or implementation.
Paper reviewed: Titans, arXiv v1 (31 December 2024)
Subsequent conference publication: NeurIPS 2025
Explore IPLEX’s work on AI technology and patent analysis.
Explore our AI and patent practice
