Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

Imagine you're developing an AI that needs to explain its decisions clearly and accurately. A common challenge is ensuring that the explanations it provides are not just plausible but also faithful to the underlying data. Current methods often fall short because they can score high on reconstruction without truly reflecting the specific claims made, leading to what's called 'structural insensitivity.' This means that even if an explanation sounds good, it might not be reliable, as it can mask inaccuracies in the model's reasoning. For instance, if a model can reconstruct an explanation well, it doesn't necessarily mean that every part of that explanation is true or relevant. This is what's called a failure mode in current evaluation techniques.

To address this, the paper introduces a new approach that focuses on the faithfulness of explanations by using a method called RECAP, which stands for Readable Encodings via Co-trained Auxiliary Predictors. The intuition here is to train additional models alongside the main one to ensure that the content being conveyed is not only reconstructible but also verifiable against independent probes. This method allows for a more nuanced understanding of what the model is actually saying, making it easier to flag inaccuracies. Compared to previous work, this approach provides a more reliable way to assess the truthfulness of explanations, which is crucial for building trustworthy AI systems that can be held accountable for their outputs.

Introduces a new method for evaluating the faithfulness of explanations in autoencoders.

Presents solid experimental results with clear metrics, though some claims lack extensive baselines.