Actions Speak Louder than Words: Measuring Cross-Lingual Policy Retention in Tool-Using Agents

Imagine you're building a multilingual AI assistant that needs to perform tasks in different languages. The challenge isn't just getting the right answer; it's about how the assistant approaches the task and whether it can adapt its actions based on the language context. Current evaluation methods often overlook this by only comparing final answers, which can hide critical failures like inconsistent actions or missteps in reasoning. This is what's called a lack of transparency in the agent's decision-making process. For instance, if an agent uses a different tool or takes a different route to solve the same problem in another language, we need to understand why that happens. Without this insight, we can't trust the agent's reliability in real-world applications.

To address this, the authors propose a new evaluation framework that focuses on the action policies of agents across multiple languages and models. They meticulously identify and eliminate five confounding factors that can skew results, ensuring that the evaluation reflects true performance rather than artifacts of the testing process. By analyzing 2.38 million rollouts, they find that while larger models maintain consistent action policies across languages, smaller models struggle, often leading to misleading conclusions. This approach not only highlights the importance of understanding agent behavior in multilingual contexts but also provides a more reliable framework for evaluating AI systems, which is crucial for developers looking to deploy these technologies effectively.

The paper introduces a new way to evaluate multilingual agent behavior by focusing on action policies rather than final outputs.

The methodology addresses confounding factors and provides a robust analysis across multiple models and languages.