LongMemEval Results
On LongMemEval-S, WOS scores from 85.2% with no LLM in the retrieval path (Tablet 1) up to 92.3% (Scroll 1.2). Five-run averages, graded by an independent model, every prompt published.
| Model | Score | σ | Reader | Delivery |
|---|---|---|---|---|
| Tablet 1zero-LLM | 85.2% | ±1.1 | Claude Opus 4.8 | Single-pass delivery. No LLM anywhere in the path. |
| Scroll 1 | 90.7% | ±0.5 | GPT-5.5 | Session-expansion delivery over the same retrieval. |
| Scroll 1.2 | 92.3% | ±0.4 | GPT-5.6-sol | Adds query expansion; time-aware delivery. |
Why these numbers are comparable
Benchmark scores for memory systems are easy to inflate: pick the best run, grade with the same model that answered, or tune retrieval to the test. We hold to one protocol and publish all of it.
- Independent judge. Answers are graded by GPT-4o, a different model from the reader, at temperature 0 under the official LongMemEval per-category rules. Nothing grades its own work.
- Five-run mean, not best-of. The full 500-question set runs five times; we report the mean and standard deviation, and publish every individual run.
- Verbatim prompts. The reader and judge prompts are published word-for-word, so the exact experiment can be reproduced.
- Nothing fitted to the wording. No part of the engine was tuned to this benchmark's phrasing, in English or in any other language.
What is LongMemEval
LongMemEval is a public benchmark for long-term interactive memory in chat assistants (Wu et al., ICLR 2025). The S configuration attaches roughly 100–140K tokens of prior conversation to each of 500 questions, spanning single-session facts, knowledge updates, preference inference, temporal reasoning, and multi-session synthesis. It is the standard we report against because it measures the hard part of memory: finding and using the right piece from a long history.
References: arXiv:2410.10813 · github.com/xiaowu0162/longmemeval
Zero LLM in the retrieval path
Tablet 1's 85.2% is reached with no language model anywhere in the retrieval path: retrieval is deterministic and semantic, and the same query returns the same memories every time. That property belongs to Tablet 1 specifically. Scroll's higher tiers add fuller delivery and query expansion to gain accuracy, which is a different trade; we report their scores without a zero-LLM claim so the two are never conflated.
Reproduce it
A third party can re-run the identical protocol at the black-box API boundary with a WOS API key and their own reader and judge keys. Each model's report page carries its per-run, per-category scores and the verbatim prompts used. Where the memory was found but not answered — the delivery gap — is analyzed in the methodology report.
FAQ
What score does WOS get on LongMemEval?
On LongMemEval-S, the WOS memory models score from 85.2% (Tablet 1, with no LLM in the retrieval path) up to 92.3% (Scroll 1.2), each reported as a five-run mean.
How is the score judged?
Answers are graded by an independent model, GPT-4o, at temperature 0, returning yes/no under the official LongMemEval per-category rules. The judge is a different model from the reader, so nothing grades its own output.
Why report a five-run average instead of the best run?
A single best run overstates what a system does in practice. We run the full set five times and report the mean and standard deviation, and we publish every run so the spread is visible.
Does recall quality change with the language?
No. It is measured at 95.2% recall@5 over 70 store-language by query-language pairs. Tablet 1 runs no LLM in the retrieval path at all.