Notes to Self: Can LLMs Benefit from Experiential Abstractions?

Imagine you're developing a large language model (LLM) to solve complex mathematical problems. The challenge isn't just about generating correct answers; it's about how the model learns from its past attempts and applies that knowledge to improve over time. Current models often struggle with this, leading to repeated mistakes or inefficient problem-solving strategies, which is what's called a lack of experiential learning. They might generate a correct answer in one instance but fail to apply similar reasoning in another, missing the opportunity to build on previous experiences.

To address this, the authors propose a method where LLMs can extract natural-language abstractions from their own solution traces or from a stronger teacher. These abstractions serve as a library of strategies that the model can reference during inference or use to enhance its training through reinforcement learning. By incorporating these experiential abstractions, LLMs show improved performance on mathematical and logical reasoning tasks. This approach not only allows models to learn from their own experiences but also demonstrates that self-extracted abstractions can be as effective as those provided by a teacher, suggesting a new avenue for enhancing LLM capabilities across various datasets and models.

The approach of extracting and utilizing experiential abstractions in LLMs is a meaningful extension of existing methods.

The findings are supported by performance improvements on established benchmarks, though more extensive validation across diverse tasks would strengthen claims.