An AI memory system scored higher than the tested full-context and retrieval alternatives when it answered questions about mixed-topic conversations, according to an arXiv preprint. At a 128k context length, TSIM posted the highest reported accuracy in each of three model tests: 69.6% with Gemma2:9b, 80.2% with Gemini 2.5 Flash and 73.8% with GPT-4o-mini. In the Gemini comparison, TSIM was 5.58 percentage points above MemGPT. The paired-bootstrap 95% confidence interval for that difference ran from 4.15 to 6.97 points. Against a tuned, non-episodic chunk-retrieval control using GPT-4o-mini, TSIM's 73.8% was 17.6 points above the control's 56.2%, and the control used 3.0k context tokens.
Finding the episode that matters
The task asks whether an assistant can recover constraints mentioned earlier, sometimes in a different subject, inside one flat, unsegmented conversation. The study tests whether it can recover those dormant, cross-domain constraints well enough to make a later task decision valid. The target is the right episode of the conversation, rather than isolated words retrieved from different places.
TSIM handles that by reconstructing an episode before assembling the answer context. Its three modules segment the thread when the subject or meaning shifts, index each episode from multiple views, then rank episodes by looking for the evidence needed for the question. The selected material is assembled into a compact answer context.
Longer prompts did not guarantee better answers
In the native full-context comparison, reported accuracy fell from 62.5% at 16k to 29.8% at 128k. TSIM remained at 73.8% at 128k while retrieving about 1k tokens. The result contrasts a compact retrieved context with the much larger constructed history shown to the Full Context system.
A separate long-window diagnostic used a 1M-token context. With Gemini, TSIM reached 96.5% accuracy against 87.2% for Full Context, using about 1.3k prompt tokens rather than 1.05M. Its reported response time was 2.16 seconds, compared with 23.87 seconds for Full Context. The diagnostic covered 400 questions, with Wilson 95% intervals of 94.2% to 97.9% for TSIM and 83.6% to 90.2% for Full Context. At 128k with DeepSeek R1, another stronger-backend check reported 93.8% for TSIM and 81.2% for Full Context.
What the benchmark can and cannot show
The component tests showed a sequence of higher reported scores as more structure was added within this benchmark. Accuracy rose from 26.2% with Standard RAG to 43.4% with fixed-token retrieval, 55.5% with semantic-drift episodes and 74.2% with the full multi-view episode stack. In an exact-evidence check, TSIM had all-evidence recall@5 of 0.810 and co-containment of 0.890, compared with 0.456 and 0.004 for Standard RAG chunks. The TSIM episodes therefore more often brought the required evidence into the top five results and kept related pieces together.
The benchmark contains 3,000 audited questions across 10 task-oriented domains, graded with deterministic four-way multiple-choice scoring. The main evaluations used constructed contexts from 16k to 128k and three answer backends: Gemma2:9b, Gemini 2.5 Flash and GPT-4o-mini. The 1M experiment was a separate diagnostic, not the main run. The format improves reproducibility and auditability, but selecting one of four answers does not show how a model handles partial or hedged replies, tool-use follow-up or long-form explanations.
Researchers also reported checks on how the benchmark was built. Deterministic machine filtering accepted 28.8% of candidates, followed by 84.3% acceptance after review by three human reviewers. The audit report lists 4,346 evidence snippets, exact full-turn records for all 3,000 questions and zero critical validation issues. A final human audit covered 300 examples, three annotators and 900 valid annotation rows. These checks document internal validation, but they cannot show that the constructed conversations match naturally occurring assistant logs.
The benchmark's synthetic origin sets a clear boundary on the result. SCALE-QA was counterfactually constructed rather than sampled from natural assistant logs, so it may not represent the distributions, styles or privacy constraints of deployed systems. The result should be read as benchmark evidence about episode reconstruction, not as evidence from naturally occurring logs or open-ended assistant use. The paper is an arXiv preprint, version 2 dated 29 September 2026.
Paper data and sources
Original title: Reconstructing the Right Episode: Evaluating Interleaved Conversational Memory Beyond Long Context
Authors: Zhexi Feng, Ruiyi Zhang, Yongbo Yang, Pengtao Xie
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text