Agent memory on Elasticsearch: hybrid retrieval and DLS - Elasticsearch Labs

Open original

This article describes an Elasticsearch-based system for long-term agent memory, dividing data into episodic, semantic, and procedural indices with tailored aging and update rules. It uses hybrid retrieval—BM25 plus dense vectors fused by RRF and refined with a cross-encoder reranker—alongside per-user DLS, supersession, decay, and an LLM-driven consolidation process to keep memory accurate and scalable.

  • Splitting agent memory into episodic, semantic, and procedural indices (each with its own decay/update rules) avoids one-size-fits-all tradeoffs in write rate and aging logic.
  • Hybrid retrieval (BM25 + Jina v5 embeddings fused via RRF, then reranked with a Jina v2 cross-encoder over 160 candidates) hit 0.89 recall@10 across 168 test queries.
  • Contradictions are resolved via supersession with time-decay ranking rather than deletion, preserving an audit trail while surfacing current facts.
  • Document-level security enforced per-user isolation with zero cross-tenant leaks, all within a single Elasticsearch cluster exposed as an MCP-compatible tool.

Elastic built an agent memory system on Elasticsearch with three dedicated indices—episodic, semantic and procedural—plus a shared catalog index. Episodic entries capture every user turn with timestamps before the AI responds. Semantic entries store stable facts about the user (“Sarah owns a Lumio Hub v2,” “Hub was reset in March”). Procedural entries hold multi-step playbooks (“troubleshoot Zigbee disconnects”) tagged with success and failure counts that update whenever the user confirms an outcome. Separating these types lets each index follow its own write rate, decay rules and update logic without bogging down the others. To fetch relevant memory, they run a two-stage hybrid retrieval: BM25 inverted-index search and Jina v5 dense embeddings fused via Reciprocal Rank Fusion (RRF), pulling 80 candidates from each leg with a rank_constant of 30. Those 160 results go to a Jina v2 cross-encoder reranker, which scores query–document pairs jointly. That mix catches literal matches—version numbers, error codes—and semantic paraphrases, avoiding the “lost in the middle” problem of large context windows. A single recall_memory call fans across all three indices plus the shared catalog; the retriever’s scoring and per-index decay automatically surface the right hits. Writes happen automatically: every user message becomes one episodic document (ID, raw text, timestamp). Elasticsearch assigns IDs and enforces per-user isolation via document-level security so no cross-tenant leaks occur. Consolidation into semantic or procedural memory runs separately: LLMs extract stable facts or refine playbooks based on success_count and failure_count. Supersession, not deletion, handles contradictions—older facts stay in the audit trail while fresh ones take priority under a time-decay ranking function. On a QA-style test of 168 queries, recall at rank 10 hit 0.89 with zero leaks between tenants. All memory types and the catalog sit in one Elasticsearch cluster, avoiding extra round-trips or multiple services. The code’s on GitHub and integrates as an MCP-compatible tool, so any LLM-based agent runtime can tap into the same memory layer without custom integrations.

Ask about this article