Where did you hear about us?
1 of 6

Research·Evaluation·November 5, 2026

Measuring what a memory engine hands over

A memory engine's score usually arrives as one number. A reader model reads the memories the engine handed over and answers, and a judge grades the answer. That number mixes what the engine handed over, how well the reader used it, and how the benchmark scores it. This is the Scroll 2 paper without the PDF, which measures the three apart.

At a glance

BEAM-1M
77.6%With memory tools, the mean of four runs
Against Tablet 2
+4.6 pointsSame reader and judge, 95% interval +3.4 to +5.9
Facts in the first twenty
78.0Coverage on fact retrieval, against 73.7 for Tablet 2
Left unused by the reader
66 of 630Questions whose memories held what the answer needed
Memory tools
57.1Memories the reader took per question, against 69.9
A larger store
+12%Text handed back as the store grew 55-fold
One question
$0.019With a million-token conversation stored
Against Claude memory
+15.6 pointsBEAM-100K, same reader, 95% interval +11.0 to +20.4

Scroll 2 on BEAM-1M

Scroll 2 scores 77.6% on BEAM-1M with memory tools, 700 questions over 74,630 conversational turns. Tablet 2, measured with the same reader and judge, scores 71.3%. Those are means of four and five runs; by the benchmark authors' own scoring they are 73.18 and 67.49. Apart from the sleep experiment, no language model processed the stored conversations, and only the reader saw memories.

What does a score mix?

Two of our own runs made the mixture concrete. With the evidence the engine handed over held fixed, changing only the reader from Claude Opus 5 to Claude Opus 5.5 left the total where it was (0.0 points) and moved summarization by −7.0. Changing only how one ability is scored splits the same Scroll 2 runs into 76.20 and 72.10: most published BEAM figures grade event ordering by rubric, and the benchmark's authors score it by rank correlation.

So every run carries three numbers. The score, computed the way the benchmark's authors compute it, with the other reading beside it. Evidence recall, the share of the turns the benchmark marks as evidence that the engine handed to the reader, with no language model involved. And Coverage: a judge reads only the memories the reader received and decides, rubric item by rubric item, whether they hold what the answer needs. The judge never sees the answer, and the reader and the judge are always different models.

BEAM-1M with the same reader and judge 700 questions · reader Claude Opus 5 · judge gpt-4.1-mini · five runs each, memory tools four
020406080Score (%)67.49Tablet 272.10Scroll 273.18Scroll 2+ memory toolsEvent ordering by rank correlation

Paired per question against Tablet 2, Scroll 2 is +4.6 points by BEAM's scoring (95% interval +3.4 to +5.9) and +4.9 by rubric; with memory tools, +5.7 and +6.2. The rubric reading sits about four points above BEAM's own for every system, which is why both are given.

How much better than Tablet 2?

With the same reader, Claude Opus 5 at maximum effort, and the same judge, gpt-4.1-mini, Scroll 2 scores 4.6 points above Tablet 2 per question (95% interval +3.4 to +5.9), and 5.7 with memory tools. The reader is Claude Opus 5 because Tablet 2 was measured with it. A new engine measured with a newer model leaves open which of the two improved.

It is ahead on nine of ten abilities. The largest gaps are summarization (65.4 against 52.5) and multi-session reasoning (73.7 against 63.8), the two that need memories from several sessions at once. The exception is temporal reasoning, 51.2 against 54.8.

What changed?

The two engines differ less in what they hold than in where they put it. On the 140 fact-retrieval questions, counting all of the roughly seventy memories handed over, Coverage is 84.6 and 84.2, nearly the same. Counting only the first twenty, it is 78.0 and 73.7: Scroll 2 puts the same facts higher. On questions that gather what is spread across sessions the gap remains even at seventy, 10.3 points on multi-session reasoning and 9.6 on summarization.

Tablet 2 holds more in one place. On knowledge update, counted to the end, it is 89.3 against 85.4, and Scroll 2 still scores higher there (81.5 against 75.4). Coverage asks whether the latest value is present, not whether an outdated one is present too, so it does not explain that difference.

What the handed memories hold Coverage · judged from the memories alone, never from the answer
0255075100Coverage (%)73.7Tablet 2first 2078.0Scroll 2first 2084.2Tablet 2all, about 7084.6Scroll 2all, about 70Information extraction and knowledge update

Counted to the end, the two engines hold nearly the same facts. Within the first twenty, Scroll 2 holds more: +4.3 points on fact retrieval and +5.9 on all questions. Per question it holds more on 157, less on 93 and the same on 375.

What did the reader leave unused?

Coverage beside the score shows where the loss happens. In 66 of 630 questions the memories held what the answer needed and the answer still fell short. Had the reader used every one of them, the score would rise from 76.8 to 82.5.

Temporal reasoning has Coverage 67.5 and scores 53.0: the judge found what the answer needs in about two thirds of the items, and the reader computed the answer less often. The reader did not see the date each memory was stored with, though dates written inside the text remained, so part of this loss may belong to the format. Preference following runs the other way, Coverage 40.0 and a score of 91.1. The rubric grades the form of the answer, and a careful answer often meets it without knowing the preference.

What if the reader decides how much to fetch?

Memory tools hand over the first page of twenty and leave the rest to the reader: the next page, the conversation around one memory, or a new search in its own words. The reader took 57.1 memories per question on average, fewer than the 69.9 of the single call, from 34 on knowledge update to 102 on summarization. The total rose by 1.1 points, with an interval that crosses zero (−0.2 to +2.3).

The largest loss is on knowledge update (−5.9), where the reader took the fewest memories and may have stopped before the latest value; we have not checked that question by question. BEAM asks one question per conversation, so what the tools are built for, carrying what was found in one turn into the next, is not exercised here.

What does sleep change?

Sleep is an experimental feature of Scroll 2. Between conversations, a language model the application chooses reads the store and writes a few short files about the user, and when the application answers, the memory tools can open them beside the memories. Measured once on and once off, the total did not move (−0.1, 95% interval −1.9 to +1.7).

The abilities move both ways. Instruction following (+7.1), summarization (+6.3) and event ordering (+5.5) rose: the files keep an instruction the user gave once, and the reader follows it. Abstention (−7.9) and information extraction (−4.7) fell, both with intervals that reach zero: the files also carry plausible content that leads the reader to answer where the record holds no answer. Sleep cost $4.29 per million tokens of memory read with Claude Sonnet 5.5.

So we suggest it on for a personal assistant or an agent carrying a task over many days, and off for work that reuses an exact value or answers that must rest on the record, such as customer support or financial and medical records.

Does a larger store hand back more?

If an engine handed back more as a conversation grew, each question would cost more and the reader's context would fill faster. We cut 31 BEAM conversations at nine points and asked at each. As the store grew 55-fold, from 1,104 memories to 61,215, the text one question got back rose 12%, from 2,997 tokens to 3,351. With a million-token conversation stored, a question costs $0.019. The engine returned its first page in 146 ms with 79 memories stored and in 185 ms with 49,372.

Per conversation turn the difference is larger. With Claude Opus 5.5 as the reader, a turn with Scroll 2 costs 4.4 to 4.8 cents at every length measured. With the whole conversation in context, a turn at a million tokens costs 22.2 cents while the cache is warm and $5.02 once it has expired, as it has for a conversation resumed the next day. A short conversation held in a warm cache is as cheap as Scroll 2 or cheaper.

How did the price change?

A memory is stored once and recalled many times. Against Scroll 1.2's prices, storing costs 23.8% more and answering a question 24.4% less, so a bill is lower once the text recalled passes about half of the text stored, and every further recall brings it closer to 24.4% less. A conversation turn costs 9.4% less. BEAM-1M stored once and asked its 700 questions 100 times costs 20.8% less in all.

How does it compare with an agent that writes its own memory?

Claude Managed Agents can attach a memory store to an agent. The agent reads and writes files in it as it works, and the store carries over to later sessions. We measured it against Scroll 2 on BEAM-100K, 20 conversations and 400 questions, as of 14 October 2026 with that day's defaults; the feature may have improved since. The reader was Claude Haiku 5.5 in every arm, so that no result rests on a strong reader.

Scroll 2 scores 54.8 and Claude memory 39.2, a per-question difference of 15.6 points (95% interval +11.0 to +20.4), and the order holds in all four pairings of two judges and two reader instructions. The gap is widest where the answer is a detail: 77.5 against 21.0 on information extraction. Claude memory is ahead on abstention (90.0 against 58.8), on instruction following and on event ordering.

The same reader with the whole conversation in context scores 63.0, 8.1 points above Scroll 2, because 127 thousand tokens still fit its window. It reads all of it for every question, at about 8 times Scroll 2's price per question. Half the conversations did not fit the default window, and BEAM-1M's conversations reach the million-token window or pass it.

Against an agent that writes its own memory BEAM-100K · 400 questions · reader Claude Haiku 5.5 in every arm · as of 14 October 2026
020406080Score (%)21.6No memory39.2Claude memory54.8Scroll 263.0WholeconversationJudge Claude Sonnet 5.5, no reader instruction, one run

Scroll 2 is +15.6 points above Claude memory (95% interval +11.0 to +20.4), and the order holds under a second judge and a second reader instruction. At 127 thousand tokens the whole conversation still fits the window and scores 8.1 points higher, at about 8 times the price per question.

Where is it weak?

Temporal reasoning is lower than Tablet 2's (51.2 against 54.8). The loss is on the reader's side, but how much of it comes from a format that hides each memory's date we have not measured separately. Event ordering, by BEAM's scoring, is 23.9, the lowest of the ten. Beside an agent that writes its own memory, it fell behind on saying that the record does not hold the answer. And BEAM is in English, so the paper makes no claim about other languages.

Who wrote this

This is our own product and the paper was written by the person who builds it. Every number in it is self-reported. What we can offer instead of independence is the procedure: the reader, the judge, the scoring and every run are in the paper, and we release the harness, the prompts and the per-question records.

Notes

Intervals
Every interval is from 2,000 bootstrap resamples of questions, paired per question.
What the reader saw
The memory text only, without dates or speakers, the same format as Tablet 2's runs. Applications that show dates and speakers may see different results, in both directions.
Tablet 2's handed memories
Tablet 2's runs kept answers and scores but not the memory text. Its evidence recall and Coverage come from a replay of its retrieval on 28 September 2026 with the same engine and corpus, which delivered 70.06 memories per question against 70.16 in the August run.
The release build
A build with the retrieval of the release, run once with memory tools on 25 October, scored 73.80 by BEAM's scoring and 78.08 by rubric. The other numbers on this page are those of the earlier builds.

The paper

Title
Wontopos Scroll 2: Score, Evidence Recall and Coverage for a Long-Term Memory Engine
Author
Sunwoo Kim, Wontopos L.L.C.
Published
5 November 2026
Benchmarks
BEAM-1M; BEAM-100K
Readers
Claude Opus 5; Claude Haiku 5.5 on BEAM-100K
Judges
gpt-4.1-mini; Claude Sonnet 5.5 on BEAM-100K
PDF
Wontopos-Scroll-2-Report.pdf, 26 pages

Was this report useful?

We publish every run and our own conflict of interest so this work can be checked. If something here is wrong, we would rather hear it.