Building agent memory on a vector database
Checked against Vector DB’s own documentation on
The most common alternative to anything on this page is not a product. It is pgvector, Pinecone or Weaviate with a few hundred lines around it, and no vendor in the retrieval path.
For a prototype that is the right answer, and anyone who tells you otherwise is selling something. What follows is a list of what tends to get written afterwards. We sell one of the things on that list.
- 2.2×the score of a raw index on BEAM-1M67.5% against 30.7%
- 95.7%LongMemEval-Swhat you get from an index is what you build
- 2,484tokens per query at 2.2M memoriesmeasured with a tokenizer, not estimated
Where WOS comes out ahead. Where it does not is in the table below.
Vector DB and WOS
| Vector DB | WOS | |
|---|---|---|
| What each one has published | ||
| BEAM-1M | 30.7% the RAG baseline published on that leaderboard | 67.5% Tablet 2 |
| LongMemEval-S | Depends what you build | 95.7% Tablet 2 read by Claude Opus 5; 93.7% read by GPT-5.6-sol |
| Latency | Whatever your index does no vendor figure to quote, because there is no vendor | ~320ms median engine round trip; 393ms at 2.21 million memories |
| Context per query | Whatever your top-k returns | 2,484 tokens BEAM-1M at 2.2M memories with verify 3 — the run the score above comes from |
| Also published | No vendor figures the numbers are whatever your own stack produces | Latency ~320ms, languages 95.2% engine round trip; recall@5 over 70 language pairs |
| Source | the BEAM-1M leaderboard | how we benchmark |
| How it is built | ||
| Stored as | Whatever you chose to embed | Kept, not rewritten |
| A model reads it | Only if you add one | Never |
| Organised around | An index, and the filters you remember | One store per end-user |
| Retrieval | Whatever you build | Not published |
| Correcting a fact | Yours to design | supersede, with lineage |
| Licence | pgvector and Weaviate are open source | Closed source |
| Self-hosting | All yours by definition | Case by case |
| Metered by | Infrastructure, plus embedding calls | Tokens, plus a flat fee per request |
- Both columns are self-reported: a placement, not a verdict. Some rows go to Vector DB.
- Across languages is a grid of 70 store-language by query-language pairs.
What WOS does
Against a vector database the difference is not the search. Your search is probably fine. The difference is the ordering and everything around it: what comes back first, how much comes back, and what happens to a row that is no longer true.
For your AI Won is the part of the product addressed to the model rather than to you, and it carries no charge. Your agent can ask how much of its own memory has been rewritten before it leans on a fact, and the answer is the same size for a hundred memories or a hundred million. the page written for an arriving agent.
- Already orderedThe first result is the best match. Recency, importance and whether something has been superseded are already in the ordering rather than being your problem.
- A budget, not a top-kA measured budget: 2,484 tokens per query at 2.2 million memories, so the prompt does not grow with the age of the account.
- Correction without deletionsupersede makes the old statement stop winning retrieval while lineage keeps it readable, which is the piece that is hardest to retrofit into rows you already have.
- Multi-hop without a modelEngrams go further than one search: deep recall, timelines, drift correction. No model in the loop, so no per-hop bill.
- Caching and idempotency includedA cache hit bills 0.1x and any write invalidates it instantly. Every write takes an Idempotency-Key. Both are the kind of thing you would otherwise build twice.
- Calls the model makes for itselfWon is the part of the product addressed to your agent rather than to you, and it carries no charge. revisions is the first of them: the model can ask how much of a memory has been rewritten before it leans on it, and the answer is the same size at a hundred memories or a hundred million.
Where Vector DB is the better answer
- Complete control, and no vendor in the retrieval path at all
- The cheapest possible bill at small scale, since you are paying for infrastructure and embeddings and nothing else
- The ability to change any part of the behaviour, which a hosted API will never give you
- No dependency on anyone else’s uptime, including ours
- What WOS costs you. It is closed source and hosted, so running it on your own hardware is a conversation rather than a download. There is no knowledge graph, so a question that is really about validity intervals is better served elsewhere. And it is the newest name on this page, with a smaller ecosystem than the ones above it.
- What it buys you. Nothing runs a language model over what you stored, so the privacy review is one sentence rather than a discussion. A query comes back in about 320ms, and the context it hands over was measured at 2,484 tokens over 2.2 million memories. Repeated queries bill a tenth. And it behaves the same in every language: recall@5 of 95.2% over 70 store-language by query-language pairs.
Why teams look at something else
| Correcting a fact is not a delete | The old statement has to stop winning retrieval while staying readable as history, and that is the hardest thing to retrofit. |
| Prompts grow with history | A naive top-k returns more as the store fills, so the per-turn cost climbs with the age of the account. |
| Isolation has to be right on the first day | Filtering by user id works until one query is built without the filter, and that failure has no visible symptom. |
| Similarity is not ranking | Cosine distance orders by closeness, which is not usefulness: recency, importance and supersession all belong in the ordering. |
- None of these are defects. Each follows from a decision that is right for somebody.
Bringing memory over
You already have the text, which is the part that usually makes migrations painful, so this is the easiest move on the page. Create one store per end-user and write their history through bulk-store. What you can then delete is usually more than expected: everything you wrote to keep results useful, and not just the search itself, moves behind the API. What you keep is your own retrieval if it is doing something specific to your product, since nothing stops you querying both and merging.
The calls are in the documentation: create a store, bulk-store, and the rest of the endpoints. A key takes a few minutes in the console.
Questions
- What does a memory API give me that a vector database does not?
The behaviour around the search. In WOS that means results already ordered for usefulness rather than raw closeness, a context measured at 2,484 tokens over 2.2 million memories, one store per end-user as the default, and supersede so a corrected fact stops surfacing without losing its history.
- Can I just use pgvector for agent memory?
For a prototype, yes, and it is often the correct engineering decision. The question is not whether vector search works, it is whether you want to own chunking, deduplication, per-user isolation, recency weighting, a context budget and fact correction as they become necessary. Those are the parts WOS ships.
- Is it cheaper to build it myself?
At small scale, usually yes, since you pay only for infrastructure and embedding calls. The comparison changes when you count the engineering time for the parts above, and it changes again if you are the person carrying the pager for them. WOS is $2 and $3 per million tokens plus $0.0001 a request, with nothing to operate.
- Can I use both?
Yes. Nothing stops you keeping your own index for whatever it does well and calling WOS for per-user history, then merging the two into one prompt.
The other options
- Mem0 Extract facts with a model as you write. Right for you if you want conversations turned into clean facts, and the widest ecosystem.
- Zep Build a temporal knowledge graph. Right for you if your questions are about when something was true, not just whether it is true.
- Letta Let the agent manage its own memory. Right for you if you want the agent to own and edit its own memory, inside their runtime.
- Cognee Run a pipeline over documents into a graph and vectors. Right for you if what you are remembering is a document corpus, and you want to own the pipeline.
- Supermemory Bundle memory with retrieval over connected sources. Right for you if you want one API over connectors, documents and chat memory together.
- Checked against Vector DB’s own documentation on September 1, 2026.
- We quote nobody else’s prices, and use only figures each project published itself.
- Out of date or unfair? Tell us and it gets corrected.
You need the same store to work in many languages, and a per-query cost that does not drift as history grows. If that is not your case, one of the projects above will serve you better.
Three calls: create a store, write a memory, recall against it. No subscription, balance starts at $5. Get an API key or read the docs.