KnowGPO: Knowledge-Grouped Policy Optimization for Search Agents

Imagine you're building an AI that needs to navigate complex tasks, like answering questions based on a vast amount of information. The challenge lies in figuring out which decisions led to success, especially when the AI interacts over many steps and the context can be messy. Traditional reinforcement learning methods often provide a single reward at the end of a task, which can obscure the contributions of individual actions — this is what's called outcome-based reinforcement learning. When trying to improve credit assignment, existing methods often rely on surface-level observations or internal model signals, which can be unreliable in deep search scenarios where the context is long and noisy.

To address this, the authors propose a framework called KnowGPO, which focuses on the knowledge gained throughout the task rather than just the observable states. By breaking down the information needed to answer a query into smaller knowledge conditions, KnowGPO tracks the progress of a trajectory based on the knowledge it accumulates at each step. This allows for a more nuanced comparison of trajectories that share similar knowledge states, and the incremental knowledge gained serves as a reward signal. Compared to previous methods, KnowGPO consistently outperforms them by a notable margin, suggesting that this approach could lead to more effective AI systems capable of handling complex, knowledge-intensive tasks.

The approach introduces a new way to assign credit in reinforcement learning based on knowledge acquisition.

The experiments show consistent performance improvements over established baselines across multiple datasets.