Measuring memory across languages and images

A memory system has one job: given a situation, return the things worth remembering about it. The usual way to score one is to hand its output to a language model and grade the model's answer, which measures the pair rather than the memory, because a strong reader absorbs a surprising amount of retrieval error. This is the paper without the PDF: what we measured, what moves it, and what it is worth beside anyone else's number.

Tablet 2 scores 95.7% on LongMemEval-S, 500 questions, and 67.5% on BEAM-1M, 700 questions over 74,630 conversational turns stored as 2.21 million memories. The engine takes a median of 393 ms per search, and no language model of its own runs anywhere in it. Both figures are quoted with a question-sampling interval rather than a run-to-run spread: repeating either benchmark moves the score by a few tenths, and printing that as ± would claim a precision the question set cannot support.

What actually moves the number?

Hold the engine, the corpus, the retrieval settings and the judge completely still, and change one thing. Changing only how many times the caller may ask again moves BEAM-1M by 8.9 points. Changing only which model reads the retrieved memories moves LongMemEval-S by 2.0. Neither is stated in the reports we would be compared against.

The second bar is not a setting. It is the entire distance between the highest published system and ours, and one knob that nobody states is worth more than it. That is why the chart below is a placement and not a ranking.

Where does it sit beside everyone else?

Third of seven, on the numbers their authors publish. We show the comparison because leaving it out would be its own kind of claim, and we call it a placement because the chart above says what a ranking would be worth: the gap between first and us is 7.5 points, and a single unstated setting is worth 8.9.

No two rows are known to share a reader, and the re-ask budget is stated in none of them. Since one of those settings is worth 8.9 points on its own, read the ordering as where the systems sit rather than which is better. Of the systems above, only Mem0 states a token count: about 6,900 per query against our 2,484.

The comparison we can actually control is against our own engines, and it is smaller than the headline. Read by the same model, Scroll 1.2 scores 92.3 and Tablet 2 scores 93.7, a gain of 1.4 points. Those come from separate campaigns rather than paired runs and their intervals overlap, so we report 1.4 as what we measured with the reader held fixed, not as an established difference. Against Tablet 1 there is no controlled comparison at all: its 85.2% was read by a different model, and the per-question comparison we produced was withdrawn.

Can you see the retrieval on its own?

Almost. One configuration comes close enough to be worth pointing at. Switch verify off entirely and drop the reader to medium effort, so the engine gets exactly one retrieval and the model reading it is the weaker of the two we tried. It scores 93.8, which edges the stronger reader running three retrieval passes, and it hands that reader 30% fewer memories to work with. That is the closest thing in the paper to a measurement of retrieval quality with the reader held still.

What does one number across every language buy?

The claim is only worth something measured against the thing it rules out. So we built a corpus where the answer is checkable by eye — thirty images, five colours by six shapes, so that “red circle” has exactly one right answer — and crossed seven store conditions with ten query languages to get seventy cells.

BM25 was configured as strongly as we could make it: standard parameters, two tokenizers run over every cell, the better of the two reported per cell. Its median across the seventy is 0.0. The only off-diagonal cells it wins are between relatives — Japanese to Chinese, which share characters, and English to German and Spanish, where cognates give a bigram tokenizer partial overlap.

The store conditions mattered more than they looked, and getting them wrong once produced a confident, wrong answer. Our first version of this experiment mixed captioned and caption-less images in one store; the captioned ones crowded the five result slots and pushed the others out, and we concluded that text queries cannot find caption-less images at all. Zero percent. Separating the stores reversed it. We report the mixed condition as a result rather than hiding it, because real customer stores are mixed.

Does a dense system come out language-neutral?

No, and this is the control we thought mattered most, because no serious competitor retrieves images with keywords anyway. We ran two open baselines that also avoid term matching, built so that only the text side differs between them: the same image encoder, one with the original English-trained text tower and one distilled to cover more than fifty languages. Across those fourteen languages Tablet 2 averages 91.4% recall@5; what separates it from the baselines is not that average but how little it moves between languages.

Both rows retrieve from the identical stored image, so the second chart is not a vision failure: it is a text encoder that was never asked to cover Russian. Swapping in a multilingual encoder lifts the mean from 19.4% to 68.5% and is level across nine of the fourteen, then falls to 5.0% on Telugu and 6.0% on Swahili. Flat over the languages you trained on is not the same as flat.

Does delivering more keep working?

Up to a point, and then it stops paying. Turning on context expansion, which attaches neighbouring material to each retrieved memory, scores highest of the three configurations we ran. We do not present it as our result, and the chart says why.

The first 8.9 points cost 3.2× the context. The last 2.8 cost 6.7× more on top of that, and that run is preliminary: 681 of 700 questions completed. It is also not uniformly better. Three question types score worse with expansion on than without it, and they are the three where the answer depends on telling similar records apart, which is exactly what more similar records makes harder. The open question is not how much context to deliver but to which questions.

Where is it bad?

Retrieval degrades sharply in low-resource languages: 53.0% recall@5 for Swahili and 64.0% for Telugu, the lowest numbers we produce anywhere, though the strongest open baseline we could run reaches 6.0% and 5.0% on those same two. Attaching an English caption to an image makes it harder to find in other languages, by 11.4 points on average across the fourteen. That is a defect in something people can buy rather than an artifact of the test, because real stores hold exactly that mixture.

Our worst BEAM-1M type is event ordering at 23.6%, and we think it measures something other than ordering. The rubric expects ten topics; our answers give ten individual events. Both are in chronological order and both have ten items, but they sit at different levels of granularity, so the matcher fails to pair them and the correlation goes negative. We could raise the score by reshaping answers to the rubric's granularity. We did not, because that is answering to the answer key rather than to the question.

Whose limit is a weak language?

From the outside these look identical: a language scores badly, and the component we did not write is the obvious suspect. Twice we had to decide which it was, and we got it wrong once. One setting omitted on the way into a stage of our own retrieval cost 37 points of Korean accuracy on short queries while leaving nine other languages untouched. We had written down the opposite conclusion, that the limit was inherent to that stage, and held it for two weeks.

The low-resource case really was the component's. Separating the two took the same procedure both times: measure the stage on its own, with everything around it removed, and see which number you reproduce. A limit you have misattributed is worse than one you have not found, because you stop looking for it.

Who wrote this

This is our own product and the paper was written by the person who builds it. Every number in it is self-reported, and so is every number we place it beside. What we can offer instead of independence is the procedure: the settings that move the score are published, the benchmarks are public datasets, every run is in the paper, and the work can be re-run rather than believed.

Footnotes

Read the full paper on arXiv →