Agent Memory Leaderboard
FROM THE AML TEAM

The AML Blog

Ideas, research, and updates from the team building Agent Memory Leaderboard.

A closer look at agent memory.
← All posts

AML Leading Methods Explained | From Similarity to Completeness: How InvMem Retrieves Truly Useful Long-Term Memories

Spotlight15 min read

Agent Memory Leaderboard

AML
Initiated jointly by universities and research institutesUnified, open, reproducible evaluation · Open Evaluation for Memory Systems


There is no single answer to Agent Memory.

This series invites leading open-source and commercial memory systems on the Agent Memory Leaderboard (AML) to explain how they write, organize, retrieve, and govern memory in the first AML Challenge. Our focus is not the ranking itself, but the technical choices, engineering trade-offs, and applicability boundaries behind the results.






Inside the First-Place Method on AML's Open-source Leaderboard


A lightweight long-term memory approach centered on fine-grained representations, hybrid retrieval, rank fusion, and recovery of dialogue structure

GitHub repository:

https://github.com/wenxiaof345-ctrl/vanilla-rag-memory

Leaderboard:

https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard

InvMem uses Dense retrieval to understand semantic paraphrases and BM25 to retain precise facts. Weighted RRF and same-session neighboring expansion then organize relevant fragments into complete evidence.




Introduction:

Agents Need Not Longer Chat Histories,

but More Effective Memory




As agents move from single-turn QA to long-term collaboration, historical information accumulates rapidly. Putting every conversation back into context verbatim may seem straightforward, but it raises costs, adds noise, and buries crucial facts. The real difficulty is not how much to store, but how to find genuinely relevant, sufficiently accurate, contextually complete evidence in a long history when a question arises.

In the first Agent Memory Leaderboard (AML) cycle, InvMem ranked first on the Open-source Methods leaderboard with an official Full evaluation score of 45.06, rounded to 45.1 in public reporting.

Unlike approaches relying on complex generation pipelines or repeated model calls, InvMem centers on a carefully organized, lightweight hybrid retrieval pipeline. It first converts dialogue into fine-grained memories, then retrieves using both semantic relevance and lexical precision, and finally recovers the local dialogue structure around matching fragments.

More detailed results show InvMem ranking first in Procedure and Workflow Execution with 37.26 and tying for first in Experience Data Discovery and Simulation. These results suggest that the value of long-term memory lies not only in retrieving similar content, but in bringing back complete rule prerequisites, execution steps, and historical experience that can inform the current task.

Core insight: Long-term memory retrieval must do more than identify the sentence most similar to a question. It must organize rules, conditions, steps, and experiential context into a set of evidence that supports the current task.

How InvMem Defines Its Role

InvMem frames long-term memory as evidence retrieval and organization, not the generation of new knowledge during storage. Historical messages remain the source of facts. The system does not call an LLM in advance to summarize dialogue or rewrite memory into summaries that may distort its meaning. Instead, it changes the granularity of historical representations, combines complementary retrieval signals, and restores local dialogue structure at the result stage.

InvMem therefore takes historical dialogue and a current question as input and returns Top K memory evidence organized by relevance. A downstream model may generate the final answer from this evidence, but InvMem's core algorithm stops at finding the correct evidence and organizing it well. This boundary makes effects easier to attribute and lets each module be analyzed and ablated independently.




Why One Similarity Measure Is Not Enough for Long Dialogues




Long-dialogue memory has three typical gaps. First, user questions often paraphrase historical statements, expressing the same meaning in different words. Second, embeddings capture overall semantics well but may weaken precise clues such as names, dates, numbers, and proper nouns. Third, even when a retriever finds the correct sentence, it may miss the surrounding subject, references, causes, or temporal conditions.

Rather than relying on a single retriever, InvMem addresses these gaps with complementary mechanisms: fine-grained chunking reduces information dilution; Dense and BM25 retrieval cover semantic and lexical clues; Weighted RRF robustly merges heterogeneous rankings; and same-session neighboring expansion restores local context.

If a conventional vector retrieval pipeline is summarized as chunking, embedding, similarity ranking, and returning Top K, InvMem mainly changes three points: it actively protects boundary semantics during chunking, adds a lexical channel and rank-level fusion during ranking, and completes evidence using local session structure before returning results. It does not overturn the basic RAG paradigm; it targets the stages most prone to errors in long conversations.

Figure 1. InvMem's Core Algorithm Pipeline




Step 1:

Turn Long Dialogues into Searchable, Fine-Grained Memories




InvMem processes dialogue using individual messages as basic boundaries. A short message directly becomes one memory unit. Long messages are split into chunks of 320 lexical units with 40 units of overlap and a sliding stride of 280. Each memory unit is then encoded into a normalized vector by BAAI/bge-small-en-v1.5, while retaining necessary context such as role, time, and session membership.

Why chunk? A long message may discuss several topics. If represented by one overall vector, local facts can be averaged into the general theme. When a question relates to only one sentence, overall similarity may not be high enough. Fine-grained chunks make each vector focus on a narrower semantic scope, helping the retriever locate specific facts.

Why overlap? Mechanical splitting may separate a condition from its conclusion or a subject from its predicate. An overlap of 40 units places information near boundaries in neighboring memories, reducing the risk of losing essential meaning because of where the cut occurs.

What this stage solves: It prevents local facts in long messages from being diluted by the overall topic while preserving complete semantics near chunk boundaries as far as possible.

Normalized Vectors: Let Similarity Reflect Direction Alone

After BGE encoding, InvMem L2-normalizes both document and query vectors. Without normalization, vector magnitude can affect dot-product scores. After normalization, every vector lies on the unit hypersphere, and the dot product reflects only directional closeness—equivalently, cosine similarity. This avoids additional bias from vector length and enables batch similarity computation between the current user's memories and the query through matrix multiplication.

The official version encodes only the original text of each current memory fragment, without concatenating preceding or following messages into the embedding input. This preserves the independence of fine-grained representations and avoids copying the same context into many vectors. Context is not mixed into encoding beforehand; it is restored through constrained neighboring expansion after retrieval.




Step 2: Dense and BM25 Retrieval




When a question arrives, InvMem first builds a unified retrieval query. Ordinary questions enter retrieval directly. If the question includes candidate options, it concatenates the question and all options into one query so that names, places, dates, and candidate facts in those options also contribute to recall. The same query then enters Dense semantic retrieval and BM25 lexical retrieval separately.

Dense: Understand the Same Meaning in Different Words

Dense retrieval encodes the query into a normalized vector and computes dot products against memory vectors. Because both sides are L2-normalized, these dot products equal cosine similarity. Dense identifies synonymous expressions, semantic paraphrases, and implicit connections. For example, historical text saying “She started working at a hospital” may be connected to “What industry does she work in?” even without identical keywords.

BM25: Preserve Precise Information That Cannot Be Blurred

Semantic embeddings alone are not reliable enough. Names, dates, numbers, abbreviations, places, and code symbols often determine answer correctness but may be weakened in an overall semantic representation. InvMem therefore runs BM25 in parallel, strengthening signals from rare, discriminative terms while using length normalization to prevent long text from gaining an automatic advantage. The official version uses classic parameters k₁=1.5 and b=0.75.

In the formula, term frequency f(t,i) measures how strongly a query term appears in the current memory; IDF(t) measures its rarity across the memory collection; and |i|/avgdl corrects for fragment length. A proper name appearing in only a few memories is usually more discriminative than common words such as “go,” “do,” or “today.” Similarly, a very long fragment does not automatically gain an advantage merely because it contains more words.

InvMem's lexical tokenization uses the same basic rules as message chunking and normalizes case. Only memories with BM25 scores greater than zero enter the lexical ranking. Fragments with no lexical overlap therefore receive no arbitrary lexical fusion bonus.

Complementary channels: Dense asks whether the meanings are similar; BM25 asks whether the key literal facts match accurately. The channels fill each other's blind spots rather than simply duplicating recall.




Step 3:

Fuse Both Rankings with Weighted RRF




Dense cosine similarities and BM25 lexical relevance scores have different meanings and numerical scales. Adding raw scores directly would require hand-designed normalization, while score distributions also change with queries and memory collections. InvMem therefore fuses rankings instead of forcing raw scores together.

Here, 60 is the RRF smoothing parameter. The Dense ranking has weight 1.0 and BM25 weight 0.5: semantic recall is primary, and lexical recall is supplementary in the official version. RRF considers only a memory's rank in each path, so the raw scores do not need matching scales, and occasional unusually high scores have less influence.

This design also has an intuitive effect: memories ranked highly by both paths receive a stable boost, while those strong in only one path still have an opportunity to become candidates. It balances consensus evidence and complementary evidence.

A Simplified Fusion Example

Suppose memory A ranks 2nd in Dense and 5th in BM25, while memory B ranks 1st in Dense but has no lexical overlap. Under the current weights, A receives both semantic and lexical contributions, while B receives only the Dense contribution. B may still lead because its semantic rank is sufficiently high, but A is not unfairly suppressed by incompatible score scales.

This example illustrates RRF's restraint. It does not turn lexical matching into an absolute verdict, nor does it ignore exact matches. BM25's weight of 0.5 supplies an auxiliary boost, while Dense's weight of 1.0 remains dominant—appropriate because long-dialogue questions often paraphrase historical expressions. The smoothing parameter of 60 reduces abrupt differences between neighboring ranks so that fusion does not depend excessively on one or two positions in a single path.




Step 4:

Recover Complete Evidence from Matching Fragments




After fusion ranking, InvMem does not immediately take the first Top K results. It selects the top 20 fused results as high-confidence seeds and adds one preceding and one following memory unit in storage order for each seed. Neighbors must belong to the same session as the seed. After deduplication, if there are still too few results, the original fused ranking is used to fill Top K.

This uses the local continuity of dialogue. A retriever may find “Yes, I will go there next week,” while the destination and reason occur in the preceding turn. It may find a conclusion whose conditions are in an adjacent fragment. Adding same-session neighbors can recover question–answer pairs, references, causal conditions, chronology, and supplementary event details.

Neighboring expansion is not a third retriever, nor does it admit arbitrary adjacent content unconditionally. Only high-confidence seeds trigger expansion, and the scope is strictly limited to one preceding and one following memory unit in the same session. This balances contextual completeness with noise control.

The key shift: InvMem ultimately aims to return not the sentences most similar to the question, but a set of memory evidence that can jointly support an answer.

Why Top-20 Seeds and ±1 Neighbors?

The seed count determines how many highly relevant fragments receive contextual supplementation. Too few seeds leave useful retrieved fragments without surrounding evidence; too many bring neighbors of low-relevance candidates into the result set. Choosing the top 20 fused results as seeds is a compromise between recall coverage and noise control.

The neighboring window also needs restraint. One memory unit on either side usually suffices to recover adjacent question–answer pairs, subjects, and event conditions. Larger windows can easily fill results with extensive local text from a single session. InvMem therefore uses a small expansion window and, after deduplication, fills remaining slots from the original fused ranking so that context recovery does not replace relevance ranking.




A Complete Example: How the Four Modules Work Together




Consider a fictional long-term dialogue. In one session, the user says, “Mina accepted an invitation to a workshop in Berlin and plans to present the Atlas project.” Immediately afterward: “She moved her departure to Friday because she has to finish a hospital shift on Thursday.” A later question asks, “Where is she going, which project will she present, and why did she reschedule?” The answer requires the location, project name, and reason for rescheduling, with evidence spread across neighboring turns.

First, chunking stores the two messages as independent, adjacent memory units instead of compressing the entire session into one broad vector. In the Dense channel, the query about where she is going and which project she will present forms semantic links to the workshop and presentation of Atlas. In the BM25 channel, precise entities such as Atlas and Berlin provide strong lexical signals. Weighted RRF raises the rank of fragments supported by both channels.

If the high-scoring seed first matches the presentation of the Atlas project, neighboring expansion brings back the same message's Berlin workshop information and the following explanation for rescheduling. The final evidence set is no longer an isolated answer fragment: it covers Berlin, Atlas, and the hospital shift, supplying a complete basis for the downstream answer.

The example's division of labor: chunking protects local facts, Dense connects paraphrased meanings, BM25 locks onto precise entities, RRF coordinates both rankings, and neighboring expansion completes evidence across turns.




Key Technical Pipeline: From Historical Dialogue to Usable Evidence




InvMem's actual technical pipeline consists of the following six steps, connecting memory representation, retrieval, rank fusion, and context recovery to organize historical dialogue into Top K evidence for downstream tasks:

1. Fine-grained representation: Process history by message, apply 320/40 overlapping chunks to long messages, and use BGE to create normalized semantic vectors.

2. Unified query: Organize the question and candidate options into one retrieval view so that discriminative clues contribute jointly to recall.

3. Dual-channel recall: Dense establishes semantic rankings; BM25 establishes precise lexical rankings.

4. Rank fusion: Aggregate both rankings with Weighted RRF, using semantics as the primary signal and lexical matches as support.

5. Structural recovery: Start from the top 20 high-confidence seeds and add one memory unit before and after each within the same session.

6. Result organization: Deduplicate and fill from the original fused ranking to obtain the final Top K memory evidence.




How Leaderboard Results and System Design Corroborate Each Other




AML's official invitation email confirmed InvMem's first place on the first Academic/Open-source leaderboard, first place in Procedure and Workflow Execution with 37.26, and a tie for first in Experience Data Discovery and Simulation. These results apply only to the first AML Academic/Open-source leaderboard, InvMem, the corresponding evaluation suite, and that cycle's rules. They cannot be generalized into a permanent ranking or a universal conclusion across application scenarios.

Procedure and Workflow Execution:

Rules Must Be Found—and Brought Back in Full

Procedure and workflow tasks usually depend simultaneously on rule meaning, precise conditions, parameters, and step order. InvMem uses Dense retrieval to link paraphrased questions with historical rules, BM25 to strengthen exact signals such as numbers, entities, and qualifiers, and Weighted RRF to robustly fuse both rankings. When a high-scoring fragment covers only one step, same-session neighboring expansion attempts to restore adjacent prerequisites and subsequent operations. This aligns with the capabilities assessed by that subcategory, but existing leaderboard results cannot be interpreted as a definite contribution from any single module.

Experience Data Discovery and Simulation:

Recover Past Experiences as Reusable Cases

Experiential data often appears as connected conditions, actions, and outcomes distributed over multiple dialogue turns. Dense finds experiences with different wording but similar situations; BM25 preserves key details such as names, locations, times, and parameters; neighboring expansion restores local processes within a session. InvMem retrieves and organizes this analogical evidence; it does not itself perform subsequent simulation or answer generation. A downstream model completes those tasks using the returned evidence.

How the Team Interprets These Two Results

We prefer to interpret these results as support for evidence completeness as a design principle. Long-term memory should not merely rank the Top 1 similar fragment first; it should make the final evidence set cover semantically relevant content, precise facts, and necessary local context together. For rule execution and experience reuse, this completeness often matches the task's actual information needs better than single-sentence similarity.

Which Conclusions Are Justified—and Which Are Not?

From the implementation, it is reasonable to say that InvMem's fine-grained representations, dual-channel retrieval, rank fusion, and neighboring expansion correspond to the two capability results highlighted in the email. Without controlled ablations, however, we cannot claim how many points any module contributed, or extrapolate this cycle's results to untested data, tasks, or deployment environments.




Next Steps:

From a Strong Baseline to a Verifiable Research System




InvMem is an official retrieval-centered version with clearly defined boundaries. It organizes memory evidence through overlapping chunks, Dense and BM25 recall, Weighted RRF, and same-session neighboring expansion. It does not yet automatically perform memory summarization, fact extraction, conflict resolution, forgetting management, or explicit global relationship modeling. These boundaries explain the actual pipeline behind the current results and identify problems for future research.

The first step will be paired, reproducible ablations around the current strong baseline. On public data such as LoCoMo-Refined, we will separately examine chunk size and overlap ratio, lexical weights, RRF parameters, neighboring seed counts, and window sizes. Only changes that replicate reliably while other conditions are controlled will count as effective gains from their corresponding modules.

Building on this, future research may explore adaptive memory granularity, session structure modeling, temporal modeling, memory updates and conflict handling, forgetting mechanisms, and indexing for larger memory scales. These are directions awaiting validation, not existing capabilities of InvMem's official Full evaluation version. SQLite remains only the persistence and metadata layer: engineering infrastructure, not a core algorithmic contribution.

Research principle: Future improvements will be grounded in reproducible public experiments and rigorous ablations, distinguishing reasonable hypotheses from validated gains and avoiding descriptions of unfinished directions as existing capabilities.




Conclusion:

Turn Long-Term Memory from Storage Capability

into Evidence Capability




InvMem's core is not simply stacking retrieval components. It establishes targeted mechanisms for three practical weaknesses in long-dialogue memory: overlapping chunks reduce information dilution; complementary Dense and BM25 recall reduces bias from a single similarity measure; Weighted RRF addresses the difficulty of comparing heterogeneous scores directly; and same-session neighboring expansion repairs incomplete evidence context.

The pipeline ultimately shifts the goal from finding similar content to organizing usable evidence. For agents engaged in long-term collaboration, memory's value lies not in how much history is saved, but in whether crucial experience can be brought back into the current task accurately, completely, and explainably when needed. InvMem's exploration is centered on that goal.




  • Team Introduction: Yigou Intelligence Team

  • Team members: Fang Wenxiao, Wu Zhiyan, Li Xingxuan, and Liu Yuhao

  • Advisors: Guo Ting, Wang Da, and Ma Yi

The Yigou Intelligence Team studies long-term agent memory, retrieval-augmented generation, and structured knowledge organization, combining algorithmic innovation with engineering practice. Its InvMem system integrates semantic retrieval, lexical retrieval, result ranking, and context expansion to provide agents with unified, reliable memory writing and retrieval. The team will continue exploring more accurate, efficient, and reproducible agent memory techniques and advancing their application in real scenarios.

Content note: The system architecture, implementation, data distributions, and related discussion in this article were provided by the InvMem team. They represent only the team's description and understanding of its own approach, not AML's official technical position or endorsement.

AML verifies only the referenced evaluation cycle, track, division, submitted version, and official leaderboard scores.

This article is not a guarantee of effectiveness in all scenarios, a product endorsement, or procurement advice.




▉ GitHub repository:

https://github.com/wenxiaof345-ctrl/vanilla-rag-memory

▉ Leaderboard:

https://huggingface.co/spaces/agent-memory-leaderboard/leaderboard