# PaTH Attention: encoding the path between tokens

An analysis of input-dependent Householder transformations, state tracking and the measured benefits and limits of PaTH Attention.

Source: https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations

HOME / NEWS & INSIGHTS NEWS & INSIGHTS PaTH Attention: encoding the path between tokens An analysis of input-dependent Householder transformations, state tracking and the measured benefits and limits of PaTH Attention. AI 2026.10.15 published IPLEX 12 min read The paper and the technical question This commentary by patent attorney Yongdeok Kim examines PaTH Attention: Position Encoding via Accumulating Householder Transformations, by Songlin Yang, Yikang Shen, Kaiyue Wen, Shawn Tan, Mayank Mishra, Liliang Ren, Rameswar Panda and Yoon Kim, affiliated with MIT, MIT–IBM Watson AI Lab, Stanford and Microsoft. The NeurIPS 2025 paper first appeared on 22 May 2025; this analysis uses arXiv 2505.16381v2, dated 3 February 2026, released under CC0 1.0. The question is how attention can follow changes of state, such as successive assignments to a variable, rather than simply recall an earlier token. From fixed positional rotation to a data-dependent path RoPE already uses content-dependent queries and keys; its limitation here is that the positional rotation rule depends on relative position, not on intervening input. PaTH inserts transformations generated by those intervening tokens into the query–key score: k_jᵀ(H_{j+1}…H_i)q_i. The ordered matrix product can distinguish paths with the same length but different state changes. Matrix multiplication is generally noncommutative, so order matters. Softmax attention and weighted values are retained; PaTH is not a linear-attention model compressing the entire past into one fixed-size state. Householder-like steps and the forgetting gate Each step has the form H = I − βwwᵀ. A small linear projection and a width-3 short convolution generate an L2-normalized w; β is controlled by twice a sigmoid, ranging between 0 and 2. At β = 2 the operation is a Householder reflection, but the general setting also includes contraction and should not be described as always norm-preserving. PaTH-FoX combines this directional path transformation with a forgetting gate that attenuates earlier contributions. The two mechanisms serve different roles: path-dependent direction and amplitude decay. Synthetic state tracking: read the conditions with the scores In the flip-flop task, a one-layer, two-head, 64-dimensional setup reduced in-distribution error from RoPE’s 6.9% to 0%; under the 98%-ignore out-of-distribution setting, errors were 40.3% and 0.0001%. These are controlled synthetic results, not a guarantee of error-free applications. Figure 2’s MQRAR-N experiment used two layers, dimension 256, two heads, 32 key–value pairs and length 768: PaTH remained strong for N below 4 but declined at N = 4. The A5 permutation task measured the minimum layers reaching 90% accuracy. Expressive-power results explain possible representations, not automatic learnability for every task. Paper Figure 2: MQRAR and A5 state-tracking experiments Language-model training: modest scale, matched comparisons The main language-model study trained 760M-parameter models on 50B FineWeb-Edu tokens, using the Mistral tokenizer and length 4096, on eight H100 GPUs for roughly two to three days. Table 2 reports WikiText perplexity of 19.01 for RoPE, 18.03 for PaTH and 17.35 for PaTH-FoX; LAMBADA perplexity was 19.77, 16.79 and 16.23. Average accuracy across six tasks was 52.7, 54.2 and 54.2, versus 53.1 for FoX. PaTH’s gain over RoPE is 1.5 percentage points, not a universal win: FoX did better than PaTH on PIQA and WinoGrande. Paper Table 2: perplexity and commonsense reasoning for 760M models Long contexts: perplexity and retrieval are different tests Figure 3 shows PaTH-FoX and FoX remaining stable to 64K tokens on the tested corpora, while PaTH alone is reasonably stable to about 32K and then deteriorates, particularly on code. Table 3 probes retrieval and state tracking directly. RULER at 8K scores RoPE 1.3, FoX 29.5, PaTH 34.8 and PaTH-FoX 34.0; at 16K the latter two score 18.7 and 22.6. On PhoneBook, PaTH versus PaTH-FoX scores 20.8 versus 93.8 at 4K and 0 versus 66.6 at 8K. PaTH leads the two shorter LongBench-E intervals. BABILONG adds narrative reasoning and state tracking. Good long-text perplexity alone does not prove reliable retrieval. Paper Figure 3: length extrapolation on three corpora Paper Table 3: RULER, BABILONG, PhoneBook and LongBench-E scores Converting a pretrained model: the recipe matters The conversion first distills attention outputs with MSE on 100M tokens, then output distributions with KL divergence on 3B tokens. A SmolLM2-1.7B checkpoint taken before its high-quality math/code stage then receives 21B tokens with a 1:1:1 text/code/math mixture. PaTH scores 25.6 versus RoPE’s 23.1 on HumanEval and 51.3 versus 47.1 on MBPP+. These gains depend on continued training. Table 5 instead converts Qwen2.5-7B-Instruct without that additional math/code stage: GSM8K rises from 80.29 to 80.67, but MATH falls from 69.10 to 65.38 and HumanEval from 82.32 to 77.44. Conversion is not a guaranteed upgrade. Paper Table 5: gains and regressions after converting Qwen2.5-7B-Instruct Parallel execution and costs that remain A compact low-rank representation of products of transformations supports blockwise preprocessing of Q/K, scans and online softmax in a FlashAttention-style algorithm. Avoiding materialization of the entire attention matrix reduces memory traffic, but computational scaling remains quadratic in sequence length. Decoding still keeps a key/value cache and applies the newest transformation to past keys. Figure 1 benchmarks Triton implementations on one H100, batch 32, 32 heads and head dimension 64: PaTH is slower than RoPE/FlashAttention but faster than FoX in that setup. It does not demonstrate universal speed superiority or elimination of the cache. Paper Figure 1: Triton implementation runtime on an H100 Limitations and the patent-analysis perspective BF16 arithmetic can become unstable near β = 2, requiring control such as clipping. The speed study uses head dimension 64; larger heads may increase compute and memory costs. A single Householder-like family does not itself contain the full family of RoPE rotations. The contribution is the combination of input-conditioned path transformations, state-tracking capability and an executable parallel algorithm, with effects demonstrated under specified conditions. Those technical distinctions do not establish patentability or infringement by themselves. PaTH and PaTH-FoX, favorable scores and regressions, numerical precision and cost must all be read together. Related service: AI patents PaTH Attention paper, version 2 Read the Korean source This article reflects the information available when it was published. Contact us to discuss your circumstances. Discuss this topic ↗ All articles ON THIS PAGE The paper and the technical question From fixed positional rotation to a data-dependent path Householder-like steps and the forgetting gate Synthetic state tracking: read the conditions with the scores Language-model training: modest scale, matched comparisons Long contexts: perplexity and retrieval are different tests Converting a pretrained model: the recipe matters Parallel execution and costs that remain Limitations and the patent-analysis perspective TALK TO IPLEX Discuss your IP questions We consider your technology and business needs together. ↗ Contact us Newer Alloy patents: why different numerical ranges may not establish inventive step ↗ Older Trade mark registration in Türkiye: filing, opposition and use ↗ Related insights AI 2026.10.14 Huawei Pangu Weather: linking forecast models across time intervals A technical reading of Huawei’s US20250258318A1 publication: selecting forecast intervals, passing predicted states and combining weather data. ↗ Read article AI 2026.10.13 LaCT: updating an AI model’s temporary memory in large chunks A technical review of Large Chunk Test-Time Training, its fast weights, GPU efficiency, causal processing and task-specific experimental limits. ↗ Read article AI 2026.10.12 Adobe’s AI image-editing patent: keeping originals and generated alternatives in layers An explanation of US12462456B2: input masks, subject masks and generative layers that preserve alternatives and editing context. ↗ Read article

- https://www.iplexlaw.co.kr/en
- https://www.iplexlaw.co.kr/en/blog/category/ai
- https://www.iplexlaw.co.kr/en/ai-patents
- https://arxiv.org/abs/2505.16381v2
- https://www.iplexlaw.co.kr/ko/blog/path-attention-householder-transformations
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-0
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-1
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-2
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-3
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-4
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-5
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-6
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-7
- https://www.iplexlaw.co.kr/en/blog/path-attention-householder-transformations#case-1837-en-8
- https://www.iplexlaw.co.kr/en/contact
- https://www.iplexlaw.co.kr/en/blog/brunch-case-1838
- https://www.iplexlaw.co.kr/en/blog/turkiye-trademark-filing-guide
- https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models
- https://www.iplexlaw.co.kr/en/blog/huawei-pangu-weather-time-interval-models
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training
- https://www.iplexlaw.co.kr/en/blog/lact-large-chunk-test-time-training
- https://www.iplexlaw.co.kr/en/blog/adobe-non-destructive-generative-editing
- https://www.iplexlaw.co.kr/en/blog/adobe-non-destructive-generative-editing