Value-Rich, Differently Organized: Measuring Value Expression and Organization in AI-Generated Debates

Imagine you're developing an AI that can engage in debates, helping users explore different viewpoints. While current evaluations focus on how well arguments are made, they often overlook how values are expressed and structured within those arguments. This can lead to a misunderstanding of the debate's depth and the effectiveness of the AI's reasoning. For instance, an AI might generate arguments that touch on many values but fail to organize them coherently, which can confuse the audience or dilute the message. This is what's called value organization failure, where the presence of values doesn't translate into effective communication.

To address this, the authors propose a framework that assesses both how values are expressed and how they are organized throughout a debate. Grounded in Schwartz's Basic-10 value theory, this framework allows for a nuanced analysis of debates by breaking down value expression and organization into measurable components. The results indicate that while AI-generated debates can invoke a wide range of values, they often lack clarity in how these values are presented across different turns. This insight suggests that for anyone building debate systems, it's crucial to not only ensure a variety of values are represented but also to focus on how these values are structured to enhance clarity and impact.

The framework introduces a new way to evaluate value organization in debates.

The findings are supported by multiple generation systems and clear metrics.