
An AI assistant can receive the right policy document and still miss the answer. Liu and colleagues varied the position of the answer-bearing document while keeping the question and document set fixed, finding poorer use of evidence in the middle for several tested models. A hypothetical travel-expense question shows how to evaluate document quantity and order separately.
Receiving evidence is different from using it
Suppose an employee asks what to attach when claiming a train fare. Search returns the expenses policy alongside approval and accounting procedures, yet the assistant explains an unrelated process. Before adding more documents, inspect what was actually supplied to the model.
Input context comprises the instructions, question and reference materials used for an answer. The context window limits the number of tokens, the units into which a model divides text. Capacity alone does not establish that evidence is used equally well throughout that input.
This discussion follows the final version of Nelson F. Liu and colleagues’ Lost in the Middle, published in Transactions of the Association for Computational Linguistics in 2024.
Changing position while holding the evidence constant
The controlled task used 2,655 NaturalQuestions-Open queries answerable from Wikipedia paragraphs. Each input contained one answer-bearing passage and retrieved, relevant distractors without an annotated answer. Related distractors make the task more demanding than a set of wholly unrelated documents.
To test position, the researchers moved the answer document within the same set. Separate length comparisons used 10, 20 or 30 documents by changing the number of distractors. The expected answer to each question remained unchanged.
Accuracy measured whether the output contained any annotated answer. That establishes answer inclusion, not whether the model cited the correct supporting passage; evidence attribution needs its own assessment.
A hypothetical expenses-policy comparison
Assume a fictional company requires a railway ticket receipt for reimbursement. Supply its evidence policy, room-booking instructions, travel approvals, accounting deadlines and staff contacts. Only the evidence policy contains the requested answer.
Place that document first, in the middle and last in three inputs, keeping the question, instructions and other documents unchanged. The expected answer is always “railway ticket receipt.” Run each version using the actual service model and settings.
If middle placement repeatedly produces an approval-procedure answer, the records identify a position worth investigating. One failed response does not establish a systematic effect: repeat the comparison across questions and supporting passages.
The paper’s twenty-document result
Figure 1 shows GPT-3.5-Turbo version 0613 with 20 documents, approximately 4,000 tokens. Approximate values read from the plot are 76% with the answer document first, 54–55% near the middle and 63% last: performance falls and then recovers.
Table 1 reports 56.1% without documents and 88.3% with only the answer-bearing document for that model. Some middle placements therefore performed below the closed-book baseline. Supplying the evidence and successfully using it should be measured separately.
These figures concern the paper’s questions, model versions and prompts. The ordinary and 16K versions of GPT-3.5-Turbo showed similar positional curves with the same 10- and 20-document inputs. They do not establish the performance of today’s models or every long-context application.
Key–value lookup and actual retrieval
A separate task required copying the value associated with a specified key from randomly generated identifier pairs. It reduced the burden of interpreting prose. Some models retrieved reliably; others lost accuracy with greater length or middle placement.
The actual Wikipedia retrieval experiment differed from the controlled task: its retrieved set could contain no answer-bearing document or several. The researchers separately measured whether retrieval found an answer and whether the reader produced it.
Retrieving more documents increased answer coverage, but answer accuracy did not rise proportionately. Additional evidence, the order of presentation and the total input length therefore deserve separate evaluation.
Records that help diagnose an internal search service
Keep the question, expected supporting passage and the exact list and order sent to the model. Missing evidence points toward query processing, indexing or ranking; evidence present but unused supports controlled comparisons of ordering or repeated passages.
Include questions requiring multiple documents. A deadline in one policy and the necessary receipt in another must both survive any reduction. Evaluate missing evidence and the accuracy of citations alongside document count and positional accuracy.
For a patent-oriented technical description, record newly developed rules for ordering candidates, grouping interdependent passages and comparing generated citations with source evidence. Holding the question and model fixed helps show which processing change reduced which failure.
The original figure labels and forms are preserved; the text explains the source material.
Reported percentages concern the paper’s historical experiments; plot readings are approximate. The company-policy example is fictional.
Baidu’s RAG approach to conflicts between model answers and retrieved knowledge
Google’s document-understanding patent: reading the layout of headings, text and images together
Explore IPLEX’s AI projects and consultation materials for documenting retrieval, evidence ordering and answer-grounding methods.
Explore our AI and patent practice
