What FID Hides: Detecting, Ranking, and Diagnosing Deviations in Generative Evaluation

Imagine you're developing a generative model for images, and you want to ensure that the outputs are not just statistically similar to real images but also visually recognizable. Currently, many people rely on metrics like Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) to gauge how well their models perform. However, these metrics can be misleading; for instance, they might indicate that a model is performing well even when it produces images that look nothing like the real thing. This is what's called a failure of moment-based metrics, where they can miss crucial distributional differences, leading to a false sense of security about the model's quality. In some cases, models can achieve low FID scores while generating images that are visually unrecognizable, which is a significant concern for builders aiming for high-quality outputs.

To address these shortcomings, the paper introduces ZID (Z-resolved Integrated Diagnostic), a new evaluation method that provides a more nuanced view of generative model performance. Instead of relying on a single scalar value, ZID offers three linked outputs: a ranking index for how much a model's output deviates from real data, a permutation p-value to test for distributional equality, and a signed readout to diagnose the nature of dispersion changes. This approach allows builders to better understand the performance of their models, especially in cases where traditional metrics like FID might fail to capture important issues like mode collapse or diversity collapse. By providing a more comprehensive diagnostic tool, ZID helps ensure that the generative models being developed are not only statistically sound but also visually coherent.

ZID introduces a multi-faceted approach to evaluating generative models, addressing limitations of existing metrics.

The experiments demonstrate ZID's effectiveness across various scenarios, though further validation may be needed.