AI memory benchmark: the engine under Second Brain OS finds the right conversation 93% of the time

You tell an assistant something in March. In September you ask about it, and it has no idea. Most AI assistants are the model. When the session ends, the context goes with it, and you tell it again.

The memory engine under Second Brain OS exists so that stops happening. Until now the only proof of how well it works was ours: a private test over our own history, scored by us. That number is real, and it is useless to you, because you cannot run it. So we ran the engine on the public test.

What the test is

LongMemEval is the public benchmark for AI memory, published at ICLR 2025 by researchers at UCLA and Tencent AI Lab. Each question comes with its own private history: about 500 chat sessions between a user and an assistant, spread across months, more than a million words. Somewhere in that pile, one to five sessions hold the answer. The benchmark’s authors have marked which ones.

The memory gets the whole history and the question. It hands back the sessions it would pull up. Then the list is checked against the known answer.

1. input500 sessionsone question's private history, indexed cold
2. The questionsent to the engine as written, nothing added
3. Top 10 sessionswhat the engine would hand a reader
4. outcomeScoreshare of the answer-holding sessions that made the list

That is the whole test. No AI judges whether an answer sounds right. The session that holds the answer is in the top 10 or it is not.

The result

1. FCT memory engineno model at index time
   93.3%
2. Contriever + fact extractionLlama 3.1 8B over every session
   86.2%
3. Stella V5 1.5B + fact extractionLlama 3.1 8B over every session
   86.2%
4. Contrieverno model at index time
   82.3%
5. Stella V5 1.5Bno model at index time
   79.4%
6. BM25keyword search
   71.0%

The engine found 93.3% of the conversations that mattered. The best published result is 86.2%. To reach it, the benchmark’s authors ran a Llama 3.1 8B language model over every one of the 500 sessions before indexing, pulling out facts for the search to find later. Their plain retrievers, with no model doing that work, land between 71% and 82%.

The same picture with a shorter list, the top 5:

1. FCT memory engineno model at index time
   89.0%
2. Contriever + fact extractionLlama 3.1 8B over every session
   76.2%
3. Stella V5 1.5B + fact extractionLlama 3.1 8B over every session
   73.2%
4. Contrieverno model at index time
   72.3%
5. Stella V5 1.5Bno model at index time
   72.0%
6. BM25keyword search
   63.4%

This wasn’t a small test. 470 questions, and even at the low end of the error range we finish above the best published result. And it ran the way it ships: your question goes in as you typed it, nothing rewritten, nothing prepared ahead of time. What we measured is what you get.

Where it is strong, and where the test is hard

The benchmark sorts its questions by what they ask of a memory. A question that points at one conversation is the easy case. A question that needs the memory to notice a fact was updated later, or to gather five separate conversations and hand back all of them, is the hard one.

1. Single conversation, assistant said it56 questions
   100%
2. Single conversation, user said it64 questions
   100%
3. A fact that changed later72 questions
   97.2%
4. Several conversations at once121 questions
   91.5%
5. Reasoning about when127 questions
   88.1%
6. A preference never stated outright30 questions
   86.7%

When the answer lives in one conversation, the engine finds it every time. When a fact was stated and then changed, it finds the conversation holding the current one 97% of the time. That is the case your business cares about most: the price that was updated, the contact who was replaced, the decision that was reversed. The hardest question is the preference nobody stated outright (“what would you suggest for my weekend?”), where the memory has to work it out from everything you have ever said. It still finds 87%.

Why it works without a model in the loop

The published methods that came closest spent a language model on every session before search could begin: read it, pull out the facts, index those. It works. It is also expensive, and it has to be redone every time the extraction model changes its mind about what counts as a fact.

The engine under Second Brain OS takes the other road. It indexes the writing as written, in pieces sized to how people talk. Then it searches two ways at once: by meaning, with a small model that runs on the laptop’s own Neural Engine, and by the exact words. The two lists are merged. You get the passages with their place in the source file, so the follow-up is a jump to the line, not a re-read of the document. Nothing in the index depends on a large model being right about anything.

What this is, and what it is not

It is a checkable number. The split, the question count, the metric, the confidence interval and the date are all above, and the run payload names the engine build that produced it. A technical advisor can reproduce it.

It is a comparison against every method the benchmark’s authors published, on their own split, at their own granularity. Several commercial memory products state results on this benchmark, and none of them publishes a table we could reach, so we claim nothing about them.

It is a measurement of the memory, not of the answer. What a language model does with the right five conversations is a separate step with a separate score. We measured the part we build.

Where this stands today

The engine is the retrieval layer inside Second Brain OS and the Custom AI Harness, delivered as installs scoped to a client, running on your own machine. It is not a standalone product, and no date is being given for one.

Six months from now, the thing you told your assistant in March should still be there when you ask. On this test, 93 times out of 100, it is.