Textual Memory
Long-context, persona, script, and conversation-memory benchmarks.
THE JOURNAL
15 postsLatest stories
48 Hours Left: CSIG AML Cycle 2 Project Submission Deadline Approaches
Key dates for Full evaluation, resuming existing tasks, participant and team registration, result review, and the official Cycle 2 leaderboard release.
AML Is in Full Swing: Participant FAQ, Part 2
Practical answers on application follow-ups, version registration, cross-track integration, endpoint issues, and model eligibility for Cycle 2.
AML Phase II Is Open: Participant FAQ
AML Cycle 2 public evaluation opened on September 20, 2026, at 00:00. This FAQ covers registration, models, Add/Search integration, recovery, and results.
Two Days to Go | CSIG-Hosted Agent Memory Challenge 2026 Cycle 2 Participation Guide
Two Days to Go | Agent Memory Challenge 2026 Cycle 2 Participation Guide
AML Official Analysis II | Coding Memory Leaderboards: How Do Industry and Open-Source Methods Perform?
Compared with textual memory, Coding Memory asks a more direct question: can experience accumulated during previous development actually help an Agent complete subsequent software engineering tasks?
AML Leading-Method Spotlight | FlowGrid AML Retriever: How a Markdown Decision Log Became an Agent Memory Retriever
There is no single answer to Agent Memory.
AML Leading-Method Spotlight | ChronoHybridMem: From Conversations to Verifiable Memory Evidence
There is no single answer to Agent Memory.
AML Leading Methods Explained | ActiveMemoryIndex: Why Does a System Without Memory Governance Rank First in Governance?
ActiveMemoryIndex explains its raw-record-first design, transparent ablations, the limits of item-count budgets, and the next steps for neighboring context and temporal reasoning.
Hosted by CSIG | RMB 150,000 Prize Pool: The Second Agent Memory Challenge Is Coming
The second Agent Memory Challenge will open on September 20, 2026, with Textual, Coding, and Multimodal tracks, unified evaluation, and an RMB 150,000 prize pool.
AML Leading Methods Explained | ReFind: Active Search Lets Agents Browse Original Chat Logs
ReFind preserves raw chat logs and moves memory computation to query-time active search. Its AML results and the InMind benchmark reveal both the strengths and limits of this approach.
AML Leading Methods Explained | From Similarity to Completeness: How InvMem Retrieves Truly Useful Long-Term Memories
InvMem combines fine-grained chunking, Dense and BM25 retrieval, Weighted RRF, and same-session neighboring expansion to turn relevant fragments into complete memory evidence.
AML Official Analysis, Part I: What Does the Textual Open-Source Leaderboard Reveal?
The top three are separated by just 0.22 points, yet excel in different ways. A closer look at seven capability areas reveals distinctive strengths, weaknesses, and open questions beyond the overall ranking.
The First AML Leaderboard Is Officially Released!
The first AML leaderboard publishes textual-track results from 67 memory frameworks, evaluated and reviewed through a unified pipeline. Explore rankings, capability profiles, and opportunities to contribute.
An Agent Only Starts to Understand You When It Has a Yesterday
Agents need more than the ability to think: they need the ability to remember. AML establishes a shared standard for comparing memory systems and invites the community to the first Agent Memory Challenge.
The Agent Memory Challenge Launches: The First Open Tournament for Memory Systems
AML brings multi-source data, a unified evaluation pipeline, capability profiles, and low-barrier integration together in an open, reproducible evaluation framework. The first Agent Memory Challenge is now open.
No posts in this category yet
AML Official Analysis II | Coding Memory Leaderboards: How Do Industry and Open-Source Methods Perform?
Agent Memory Leaderboard
Compared with textual memory, Coding Memory asks a more direct question: can experience accumulated during previous development actually help an Agent complete subsequent software engineering tasks?
The first cycle of the Agent Memory Leaderboard (AML) Coding Memory evaluation was built around 12 real code repositories. It assembled 1,290 annotated historical engineering tasks and set 150 tasks to be solved, including 51 New Feature tasks and 99 Bug Fix tasks.
The Memory system writes and retrieves relevant information from historical tasks, then passes it to a unified Coding Agent to complete the current task.
The first-cycle results reveal two very different patterns: a clear gap has emerged at the top of the Industry leaderboard, while Open-source Method scores remain tightly clustered.
Industry Leaderboard:
The Leader Is 10 Percentage Points Ahead of the Second Tier
On the Industry leaderboard, MemoraX ranks first with a Task Solve rate of 62.00%, comprising 70.59% for New Feature and 57.58% for Bug Fix.
Next, claude-mem, hs, and MemOS all achieve an overall Task Solve rate of 52.00%. In other words, MemoraX has established a 10-percentage-point lead over the second tier.
Public information shows that MemoraX Code does not treat Memory merely as simple storage and retrieval. It also considers which information is worth writing, which memories need updating, and when they should be invoked again.
MemoraX Code combines local code-repository memory with long-term memory and uses a Memory Model to learn memory selection, updating, and invocation. On top of historical development trajectories, the system further distills Procedure Memory, turning previously validated analysis, execution, and verification workflows into engineering experience that subsequent tasks can reuse.
In one set of public experiments, MemoraX distilled 15 pieces of engineering experience from 123 historical task segments, then further organized them into 4 categories of Procedure Memory.
Rather than simply preserving what happened in the past, this approach focuses on a different question: which past experiences are worth using again during the next development task?
claude-mem, by contrast, places more emphasis on continuously recording the development process and restoring it on demand.
It records tool calls and Observations produced while a Coding Agent works, organizes them into more compact semantic records, and persists them. When a new Session begins, the Agent can first search for relevant records, inspect the surrounding development process through a Timeline, and finally read only the complete content that the current task actually requires.
This layered retrieval avoids putting large amounts of history back into the Context all at once, while preserving the ability to trace backward and forward through the complete development record.
On the leaderboard, claude-mem achieves an overall Task Solve rate of 52.00%, with 56.86% for New Feature and 49.49% for Bug Fix.
The Industry leaderboard shows that different product directions are beginning to emerge in Coding Memory. Some focus on continuously recording and restoring the Agent's development trajectory; others go further, trying to select and distill genuinely reusable experience from long-term development.
In the first-cycle leaderboard, these differences have already begun to show up in the final success rates of software engineering tasks.
Open-Source Leaderboard: 8 Methods Tie for First Place
By comparison, the leading scores on the Open-source Methods leaderboard are much more concentrated.
AM-Link, AMC-Memory, aml-memory-baseline, aml-memory-mvp, causal-memory, Hybrid Episodic Memory, Memoria, and nano-memory all achieve a Task Solve rate of 52.67%, with 8 methods tied for first place.
Immediately behind them, agent-memory, ChronoHybridMem, mem0-BQE, and MemoryBear also reach 52.00%. Across the dozen or so leading methods, the final scores differ by less than 1 percentage point.
With scores distributed this way, dwelling on first, second, and third place is no longer particularly meaningful. A more interesting question is what different Memory approaches these similarly scoring methods actually use.
Here we examine three representative implementations.
The causal-memory submission primarily preserves raw historical information. At retrieval time, it uses both lexical matching and semantic retrieval, then fuses different signals to rank historical content relevant to the current task.
The overall idea is straightforward: avoid complex abstraction of the history in advance, preserve as much raw information as possible, and focus on effective retrieval.
Ultimately, causal-memory achieves an overall Task Solve rate of 52.67%, with 62.75% for New Feature and 47.47% for Bug Fix.
Memoria also uses hybrid retrieval, but makes additional optimizations for Coding scenarios.
Besides semantic retrieval and full-text keyword search, it pays particular attention to function names, file paths, snake_case, CamelCase, exception messages, and code Tokens, and supplements the context with historical messages near matching records.
This allows very specific engineering signals, such as function names, paths, and error messages, to participate directly in retrieval without relying entirely on the semantic similarity of natural language.
Memoria likewise achieves an overall Task Solve rate of 52.67%, comprising 60.78% for New Feature and 48.48% for Bug Fix.
agent-memory takes a more direct Raw-First approach.
Historical messages are not first summarized into new Memory by an LLM; they are stored directly as raw records. Retrieval then combines BM25 with local Dense Retrieval and fuses the different results.
Its overall Task Solve rate is 52.00%, with 50.98% for New Feature and 52.53% for Bug Fix.
At least from the current results, more complex preliminary abstraction of history is not a prerequisite for good performance. Preserving raw engineering records is also an effective route.
Looking separately at New Feature and Bug Fix makes the differences between methods more apparent.
The respective rates are 62.75% / 47.47% for causal-memory, 60.78% / 48.48% for Memoria, and 50.98% / 52.53% for agent-memory.
These scores alone cannot establish that a particular Memory design is inherently better suited to a particular type of task. They do show, however, that although both new-feature development and bug fixing need historical experience, the information that is genuinely useful may differ.
When developing new features, an Agent may rely more on previous implementations, module relationships, and project conventions. During bug fixing, error messages, modification paths, failed attempts, and specific fix records often matter more.
How to organize and retrieve historical experience for different software engineering tasks remains an open problem for Coding Memory.
The first-cycle Coding leaderboard does not yet show that any particular technical approach has become the definitive answer. The highest Industry Task Solve rate is 62.00%, and the leading Open-source tier is at 52.67%. Both commercial systems and open-source methods still have substantial room for improvement.
Coding Memory has nevertheless taken memory evaluation a step further: the focus is no longer only on how much history an Agent stores or how much content it retrieves, but on whether that historical experience ultimately improves completion rates on real software engineering tasks.
Memory is moving from remembering the past toward reducing repeated trial and error.
The Second Agent Memory Challenge Is Coming Soon
The second Agent Memory Challenge, an open challenge for Agent memory, will launch soon, with registration opening on September 20, 2026. Please see the announcement article for details. Eligible universities, research institutions, companies, open-source teams, and individual researchers are welcome to participate and help build and advance the Agent Memory Benchmark!
Hosted by CSIG | A RMB 150,000 Prize Pool: The Second Agent Memory Challenge Is Coming Soon