What remains after retrieval saturates

Retrieval is solved. The answer still misses.

The engine surfaced the right memory 99.6% of the time, and that number barely moves whether it returns five candidates or twenty. Finding a memory in a long history is, for this benchmark, close to solved.

And yet end-to-end accuracy sits well below that. The memory was right there, and the answer still came out wrong 9 to 15 times in a hundred. We call that distance the delivery gap: the share of questions where the memory was found but never turned into a correct answer. The rest of this report is about what lives inside that gap.

How we measured it

We used the full LongMemEval-S set, 500 questions, each carrying about 53 sessions of prior conversation (roughly 140,000 tokens). Every model ran five times and we report the mean, with no run dropped and no best result cherry-picked. Answers were graded by an independent model, never the one that wrote them. Two of our models share identical retrieval, so anything separating their scores happens after the memory is found. That makes them an instrument for isolating delivery, not a leaderboard.

Where it goes wrong

The wrong answers are not spread evenly. Where the answer sits in one session, the models are already near the ceiling. Almost everything that stays wrong is concentrated in two kinds of question: reasoning across many sessions, and reasoning about time. Together they hold roughly four out of every five remaining errors.

Why it still misses when the memory is right there

We read the wrong answers one by one. They fall into four kinds, and only one of them is really the model failing to use what it was handed.

1. A limit of the ruler, not the system. A handful of answers were essentially correct, but the automatic grader marked them wrong. We keep the grading fixed and absorb that loss rather than nudge our own scores up.
2. A principle we keep on purpose. A few questions can only be reached by matching exact date strings. Our engine holds the same behaviour in every language, so those exact-string questions have nothing to latch onto. Those few stay unsolved as the price of that principle.
3. The hard core: putting the pieces together. The largest block is questions that need counting, deduplicating, and summing facts spread across many sessions. Even with every piece in context, missing one instance breaks the whole total, and all-or-nothing grading gives no partial credit. This is not a formatting problem. It is the real limit of using long context.
4. Delivery truncation, already closing. The last kind is where the memory was found but a single pass left the neighboring pieces behind. Handing over more of what was found brings them along, and this is the part the fuller model already recovers.

So most of what remains is not the model failing to find. Retrieval found it 99.6% of the time. What is left is aggregation over what was delivered, and a grading method honest enough to give partial credit. That is where we point next.

What actually moves the number

Between our two models, retrieval is identical, so the 85.2 to 90.7 difference is not a better search. It is a fuller delivery of the same search: the lighter model hands the reader about 1,200 tokens, the fuller one about 3,700, carrying the neighboring passages along. That closes the truncation channel above. What it does not close is the aggregation core, which is why a gap to the ceiling remains even for the fuller model. The next lever is not more retrieval or more delivery volume. It is reasoning over evidence that has already arrived.

The cost, and the conflict

Fuller delivery is not free. The fuller model reads about three times the tokens, so it costs more to run. That trade is deliberate: for questions where one missing piece flips the answer, completeness is worth the tokens; for questions already answered from a single session, the lighter model is enough. And one thing worth saying plainly: these systems are our own products, and this report was written by the person who builds them. That is a conflict of interest. We offset it by publishing every run, every prompt, and the full protocol, taking no best-of numbers, and leaving anyone free to re-run the same test and check us.