MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
Xixuan Hao, Zeyu Zhang, Zehao Lin, Yihang Sun, Ziliang Guo, Xichong Zhang, Yuxuan Liang, Feiyu Xiong, Zhiyu Li
Read on arXiv →
Key claim
MemOps transforms memory evaluation from answers to operations.
In plain English
Current benchmarks for evaluating long-term memory in LLMs primarily focus on the correctness of final answers, which can obscure underlying memory issues. This paper identifies that memory should be viewed as a dynamic process involving various operations like remembering and forgetting. By introducing MemOps, a benchmark that tracks these operations, the authors provide a more nuanced evaluation of memory performance. Builders might find this approach valuable for developing more reliable and interpretable memory systems in their applications.
Introduces a new framework for evaluating memory in LLMs.
Provides structured evaluation metrics and reveals critical failure modes.
Deep reliability assessment
The methodology supports a detailed analysis of memory operations in LLMs, but the claim of moving towards interpretable, operation-level diagnosis may be overclaimed without broader empirical validation across diverse settings.
Reproducibility
Yes, the paper provides a GitHub repository for the code: https://github.com/MemTensor/MemOps.
Key figure
Figure 1 compares existing long-term memory benchmarks with MemOps, highlighting the lifecycle memory operations and structured operation traces.