The Human Creativity Benchmark
Aspen Hopkins, Allison Nulty, Alexandria Minetti, Anoop Pakki, Angad Singh
Read on arXiv →
Key claim
Evaluating creative AI requires distinguishing between shared quality and personal taste.
In plain English
Imagine you're trying to build an AI that can create art, write stories, or design products. You want to know how good it is, but measuring creativity is tricky. Traditionally, evaluators might treat disagreements about what makes something 'good' as errors, but in creative fields, those differences often reflect personal taste rather than mistakes. This is where the Human Creativity Benchmark (HCB) comes in. It collects feedback from professionals in creative domains, focusing on two important aspects: convergence, where experts agree on technical quality, and divergence, where personal taste varies. By analyzing 15,000 judgments across different creative tasks, the HCB shows that while experts align on technical correctness, they diverge on aesthetic preferences. This means that if you just boil everything down to a single score, you lose valuable insights about what works and what doesn't in different contexts. For builders, this approach means you can better understand where your models need to be precise and where they can be more flexible, leading to more effective creative AI systems.
The paper introduces a new evaluation framework that distinguishes between convergence and divergence in creative AI assessments.
The findings are based on a large dataset of professional judgments across multiple creative domains, providing solid evidence for the claims.
Deep reliability assessment
The methodology supports that expert judgments separate into more convergent (checkable) vs more divergent (taste-driven) dimensions within the paper’s five domains and three workflow stages using ~15k professional ratings. It likely overclaims generality to creative work at large and optimization guidance for models, given a small, non-random rater pool, limited prompts, and uncontrolled model nondeterminism.
Reproducibility
Dataset: yes (released CSVs with prompts, model outputs, pairwise comps, scalar ratings, qualitative notes). Code: no public repo or evaluation interface code link provided; the Contra Labs rater UI appears proprietary. Some generation settings were standardized, but randomness and sampling variability are not fully controlled.
Key figure
Figure 1 depicts the creative process as a narrowing ‘sideways martini’ from broad Ideation to Mockup to precise Refinement, illustrated with example ad-image stills at each stage.