The paper and the technical question
This commentary by patent attorney Yongdeok Kim examines PaTH Attention: Position Encoding via Accumulating Householder Transformations, by Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda and Yoon Kim, affiliated with MIT, MIT–IBM Watson AI Lab, Stanford and Microsoft. The NeurIPS 2025 paper first appeared on 22 May 2025; this analysis uses arXiv 2505.16381v2, dated 3 February 2026, released under CC0 1.0. The question is how attention can follow changes of state, such as successive assignments to a variable, rather than simply recall an earlier token.
From fixed positional rotation to a data-dependent path
RoPE already uses content-dependent queries and keys; its limitation here is that the positional rotation rule depends on relative position, not on intervening input. PaTH inserts transformations generated by those intervening tokens into the query–key score: k_jᵀ(H_{j+1}…H_i)q_i. The ordered matrix product can distinguish paths with the same length but different state changes. Matrix multiplication is generally noncommutative, so order matters. Softmax attention and weighted values are retained; PaTH is not a linear-attention model compressing the entire past into one fixed-size state.
Householder-like steps and the forgetting gate
Each step has the form H = I − βwwᵀ. A small linear projection and a width-3 short convolution generate an L2-normalized w; β is controlled by twice a sigmoid, ranging between 0 and 2. At β = 2 the operation is a Householder reflection, but the general setting also includes contraction and should not be described as always norm-preserving. PaTH-FoX combines this directional path transformation with a forgetting gate that attenuates earlier contributions. The two mechanisms serve different roles: path-dependent direction and amplitude decay.
Synthetic state tracking: read the conditions with the scores
In the flip-flop task, a one-layer, two-head, 64-dimensional setup reduced in-distribution error from RoPE’s 6.9% to 0%; under the 98%-ignore out-of-distribution setting, errors were 40.3% and 0.0001%. These are controlled synthetic results, not a guarantee of error-free applications. Figure 2’s MQRAR-N experiment used two layers, dimension 256, two heads, 32 key–value pairs and length 768: PaTH remained strong for N below 4 but declined at N = 4. The A5 permutation task measured the minimum layers reaching 90% accuracy. Expressive-power results explain possible representations, not automatic learnability for every task.
Language-model training: modest scale, matched comparisons
The main language-model study trained 760M-parameter models on 50B FineWeb-Edu tokens, using the Mistral tokenizer and length 4096, on eight H100 GPUs for roughly two to three days. Table 2 reports WikiText perplexity of 19.01 for RoPE, 18.03 for PaTH and 17.35 for PaTH-FoX; LAMBADA perplexity was 19.77, 16.79 and 16.23. Average accuracy across six tasks was 52.7, 54.2 and 54.2, versus 53.1 for FoX. PaTH’s gain over RoPE is 1.5 percentage points, not a universal win: FoX did better than PaTH on PIQA and WinoGrande.
Long contexts: perplexity and retrieval are different tests
Figure 3 shows PaTH-FoX and FoX remaining stable to 64K tokens on the tested corpora, while PaTH alone is reasonably stable to about 32K and then deteriorates, particularly on code. Table 3 probes retrieval and state tracking directly. RULER at 8K scores RoPE 1.3, FoX 29.5, PaTH 34.8 and PaTH-FoX 34.0; at 16K the latter two score 18.7 and 22.6. On PhoneBook, PaTH versus PaTH-FoX scores 20.8 versus 93.8 at 4K and 0 versus 66.6 at 8K. PaTH leads the two shorter LongBench-E intervals. BABILONG adds narrative reasoning and state tracking. Good long-text perplexity alone does not prove reliable retrieval.
Converting a pretrained model: the recipe matters
The conversion first distills attention outputs with MSE on 100M tokens, then output distributions with KL divergence on 3B tokens. A SmolLM2-1.7B checkpoint taken before its high-quality math/code stage then receives 21B tokens with a 1:1:1 text/code/math mixture. PaTH scores 25.6 versus RoPE’s 23.1 on HumanEval and 51.3 versus 47.1 on MBPP+. These gains depend on continued training. Table 5 instead converts Qwen2.5-7B-Instruct without that additional math/code stage: GSM8K rises from 80.29 to 80.67, but MATH falls from 69.10 to 65.38 and HumanEval from 82.32 to 77.44. Conversion is not a guaranteed upgrade.
Parallel execution and costs that remain
A compact low-rank representation of products of transformations supports blockwise preprocessing of Q/K, scans and online softmax in a FlashAttention-style algorithm. Avoiding materialization of the entire attention matrix reduces memory traffic, but computational scaling remains quadratic in sequence length. Decoding still keeps a key/value cache and applies the newest transformation to past keys. Figure 1 benchmarks Triton implementations on one H100, batch 32, 32 heads and head dimension 64: PaTH is slower than RoPE/FlashAttention but faster than FoX in that setup. It does not demonstrate universal speed superiority or elimination of the cache.
Limitations and the patent-analysis perspective
BF16 arithmetic can become unstable near β = 2, requiring control such as clipping. The speed study uses head dimension 64; larger heads may increase compute and memory costs. A single Householder-like family does not itself contain the full family of RoPE rotations. The contribution is the combination of input-conditioned path transformations, state-tracking capability and an executable parallel algorithm, with effects demonstrated under specified conditions. Those technical distinctions do not establish patentability or infringement by themselves. PaTH and PaTH-FoX, favorable scores and regressions, numerical precision and cost must all be read together.
PaTH Attention paper, version 2