Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark

[groksrc]

Context: fresh benchmark run (2026-06-10)

Same run as #950: full LoCoMo via basic-memory-benchmarks (run 3e11241b9d56, 1,986 queries), BM main @ 0.21.6 (de53e0e) vs mem0ai 2.0.5. See #950 for headline numbers and setup.

The finding

Of the 281 queries BM uniquely missed at recall@5, 146 are true retrieval misses (gold doc absent from top-10). The dominant pattern is cross-conversation entity confusion — proper nouns in the query don't pull their weight against generic semantic similarity, so documents from the wrong conversation outrank the gold doc:

• "What are Joanna's hobbies?" — gold is locomo-c03-s01/s02; BM's top hits lead with locomo-c07-* docs (different people entirely)
• "Who is Anthony?" — gold locomo-c04-s04; BM's top 5 are spread across c06/c01/c05, none from the right conversation
• "What symbolic gifts do Deborah and Jolene have from their mothers?" — right conversation, but generic Deborah/Jolene chatter outranks the session that answers it

This concentrates in the single-hop category (entity-attribute lookups), BM's weakest: R@5 0.486 vs mem0's 0.592.

Proposal

This is where the knowledge graph should be an unfair advantage — entities are first-class in Basic Memory. Boost search candidates whose extracted entities match proper nouns / entity mentions in the query, so "Joanna" strongly prefers documents whose entity set actually contains Joanna. Possible shapes:

• a ranking boost term when query-detected entities intersect the doc/chunk's linked entities
• or an entity-filtered first pass with semantic fallback, fused with the existing hybrid score

Complementary to the rerank stage proposed in #950: reranking fixes near-misses already in the candidate set; entity boosting fixes the candidate set itself. Both measurable commit-to-commit with the benchmarks repo's worktree workflow.

🤖 Generated with Claude Code

[claude]

Claude finished @groksrc's task in 2m 7s —— View job

---

Issue Triage

• Gather context and read issue
• Classify issue type, priority, component, complexity
• Check for duplicate issues
• Add appropriate labels (enhancement applied; other proposed labels don't exist yet in the repo)
• Provide triage summary

---

Classification

Dimension	Assessment
Type	Enhancement — benchmark-driven proposal to improve search ranking quality
Priority	Medium — meaningful quality gap (single-hop R@5: 0.486 vs mem0's 0.592), but no breakage
Component	Search/MCP — search_service.py, search_repository.py, and the search_notes MCP tool
Complexity	Complex — requires query-time entity extraction, index/scoring changes, and benchmark validation

---

Not a Duplicate

This is complementary to #950, not a duplicate:

• Add a rerank stage to search: ~half of LoCoMo benchmark misses are ranking failures, and there is ~20x latency headroom #950 targets ranking failures — the gold doc is already in BM's top-10 but ranked 6–10. Fix: rerank the candidate set.
• Entity-aware ranking boost: cross-conversation entity confusion dominates hard retrieval misses in LoCoMo benchmark #951 targets retrieval failures — the gold doc isn't in BM's top-10 at all. Fix: bias the candidate set toward entity-matching docs.

Both are needed; neither subsumes the other.

---

Architecture Notes

Reading search_service.py and search_repository.py, the current search pipeline is:

1. SearchService.search() normalizes the query into _PreparedSearchQuery (text, filters, retrieval mode)
2. Passes to SearchRepository.search() — FTS, vector, or hybrid depending on retrieval_mode
3. Results returned as-is — no post-retrieval re-scoring

The knowledge graph has exactly the structure needed for entity boosting: every entity has a title (the entity name) and permalink (slugified), and docs link to each other via relations. The bottleneck is that the query side has no awareness of which tokens are entity references — "Joanna" and "hobbies" are weighted equally.

Two feasible shapes mentioned in the issue:

Option A — Ranking boost (lower risk, no schema change):
After initial retrieval, detect proper-noun-like tokens in the query (capitalized, not stop-words), look up whether each candidate doc's title or its linked entity names contain any of those tokens, and apply a score multiplier. This is a pure post-processing step on the existing List[SearchIndexRow].

Option B — Entity-filtered first pass (higher recall improvement potential):
Extract entity candidates from the query, retrieve a set of docs whose titles/relations match those candidates, then fuse this set with the standard hybrid results before re-ranking. Needs a new repository method but no schema changes if entity lookups hit the existing Entity table.

Option A is lower-blast-radius and measurable commit-to-commit; Option B is more powerful but needs more careful integration with the hybrid scoring.

---

Suggested Next Steps

1. Decide between Option A and B (or sequence A then B)
2. Implement a proper-noun / entity-token extractor for query strings — could start with a simple heuristic (tokens that are Title-Cased and not in FTS_RELAXED_STOPWORDS) before reaching for a full NER model
3. Wire entity match signal into SearchService.search() as a post-retrieval boost pass
4. Measure against LoCoMo benchmark using the benchmarks repo worktree workflow to confirm improvement to single-hop R@5

This is well-scoped and measurable — the benchmark harness gives a direct signal. The knowledge graph's entity-first design makes Option A implementable without schema changes, making it a good starting point.