When Does On-Policy Interaction Help? Representational Tradeoffs in Value-Based Imitation Learning

Imagine you're building a robot that learns to perform tasks by watching an expert. The challenge is that traditional methods, like Behavior Cloning, often struggle when the robot can't perfectly mimic the expert's actions. This leads to compounding errors and a plateau in performance, especially when the robot's understanding of the expert's strategy is limited. This issue is known as the representational gap, where the learner's model can't fully capture the expert's policy, leading to what's called policy mismatch. To address this, researchers have found that interacting with the expert during the learning process can significantly improve outcomes. By querying the expert while the robot is learning, the robot can focus on understanding the expert's value function instead of trying to replicate every action perfectly. This insight leads to the development of a new algorithm called OVI, which allows the robot to learn more efficiently by leveraging expert feedback and focusing on value estimation rather than direct policy imitation. Compared to existing methods like DAgger and offline policy-based approaches, OVI shows substantial improvements, particularly when the robot's learning model is less complex than the expert's. This means that for builders, OVI offers a more robust framework for training agents that can adapt and learn effectively from expert demonstrations.

The approach shifts the focus from policy replication to value function realization.

Empirical results show OVI's effectiveness against established methods, though some assumptions are needed.