Wasserstein Policy Gradient for Entropy-Regularized Linear-Quadratic Control

Imagine you're developing a control system for a robot that needs to navigate a complex environment while making decisions based on its current state. Traditional methods often rely on fixed policies that can struggle to adapt to changing conditions, leading to suboptimal performance or even failure in dynamic scenarios. This is particularly evident in linear-quadratic (LQ) control problems, where the assumptions about the system's behavior can lead to inaccuracies in policy execution, a situation known as model mismatch. As a result, engineers often face challenges in ensuring that their control systems remain effective under varying circumstances.

To address these issues, the authors propose a new approach called Wasserstein policy gradient (WPG), which updates action policies by considering the geometry of the action space. By incorporating entropy regularization, WPG allows for a more flexible and adaptive policy that can better handle uncertainties in the environment. The paper provides a mathematical foundation showing that WPG leads to a well-defined ordinary differential equation (ODE) for optimizing feedback gains and action covariances, ensuring that the system converges to an optimal policy efficiently. Compared to traditional methods, this approach not only enhances the robustness of control systems but also provides a clearer understanding of how policies can be adjusted in real-time, making it a valuable tool for builders working on complex control applications.

The approach introduces a new method for optimizing policies in control problems using Wasserstein gradients.

The results are backed by a rigorous mathematical framework and proofs of convergence.