The Human Creativity Benchmark
Aspen Hopkins, Allison Nulty, Alexandria Minetti, Anoop Pakki, Angad Singh
Read on arXiv →
Key claim
Evaluating creative AI requires distinguishing between shared quality and personal taste.
This paper presents the Human Creativity Benchmark (HCB), which captures both shared best practices and individual tastes in creative AI evaluation. A key finding is that models do not perform uniformly across different creative phases, highlighting the need for nuanced evaluation metrics.
In plain English
Imagine you're trying to build an AI that can create art, write stories, or design products. You want to know how good it is, but measuring creativity is tricky. Traditionally, evaluators might treat disagreements about what makes something 'good' as errors, but in creative fields, those differences often reflect personal taste rather than mistakes. This is where the Human Creativity Benchmark (HCB) comes in. It collects feedback from professionals in creative domains, focusing on two important aspects: convergence, where experts agree on technical quality, and divergence, where personal taste varies. By analyzing 15,000 judgments across different creative tasks, the HCB shows that while experts align on technical correctness, they diverge on aesthetic preferences. This means that if you just boil everything down to a single score, you lose valuable insights about what works and what doesn't in different contexts. For builders, this approach means you can better understand where your models need to be precise and where they can be more flexible, leading to more effective creative AI systems.
The paper introduces a new evaluation framework that distinguishes between convergence and divergence in creative AI assessments.
The findings are based on a large dataset of professional judgments across multiple creative domains, providing solid evidence for the claims.
Deep reliability assessment
The methodology supports that expert judgments separate into more convergent (checkable) vs more divergent (taste-driven) dimensions within the paper’s five domains and three workflow stages using ~15k professional ratings. It likely overclaims generality to creative work at large and optimization guidance for models, given a small, non-random rater pool, limited prompts, and uncontrolled model nondeterminism.
Reproducibility
Dataset: yes (released CSVs with prompts, model outputs, pairwise comps, scalar ratings, qualitative notes). Code: no public repo or evaluation interface code link provided; the Contra Labs rater UI appears proprietary. Some generation settings were standardized, but randomness and sampling variability are not fully controlled.
Discussion questions
- 1.Claim stress test: The authors argue evaluator disagreement in creative work is often legitimate taste, not noise—does that match your design reviews, or do you find that ‘disagreement’ usually signals missing requirements or poor prompt scoping?
- 2.Missing eval: They report convergence/divergence across Prompt Adherence, Usability, and Visual Appeal but don’t show inter-rater reliability by axis and stage—would high or low IRR change how you’d act on the ‘convergent’ axes (e.g., making them hard gates in a pipeline)?
- 3.Tradeoff pick: If you had to ship tomorrow, would you prefer a system that is rock-solid on convergent axes but highly variable on divergent ones (more steerable, higher risk), or a homogenized model that’s merely ‘pretty good’ across all axes (faster handoffs, less distinctiveness)?
- 4.Baseline challenge (code/UI domains): They rate ‘Usability’ without executing code or user tests for Landing Pages/Desktop Apps—should a stronger baseline include functional checks (accessibility linters, Lighthouse, runnable prototypes), and would that likely increase or decrease measured convergence?
- 5.Experience prompt (market fit): The raters come from Contra’s network; for those building in SEA markets, have you seen different aesthetic norms or brand risk tolerances that would flip what this benchmark marks as ‘divergent’? Would you reweight axes or rater personas for your locale?
Key figure
Figure 1 depicts the creative process as a narrowing ‘sideways martini’ from broad Ideation to Mockup to precise Refinement, illustrated with example ad-image stills at each stage.