CreativeInstruct: Scalably Teaching LLMs to Balance Quality, Creativity, and Diversity

Imagine you're developing a large language model (LLM) for creative tasks like story generation. The challenge is that while post-training can enhance the model's capabilities, it often leads to less diverse and creative outputs, which is a problem when creativity is essential. This is what's called a trade-off between quality and creativity, where models become more rigid and less imaginative after fine-tuning. Current methods typically focus on improving performance metrics but overlook the need for creative expression in generated content. To address this, the authors propose a method called CreativeInstruct, which teaches LLMs to maintain a balance between the creativity of the base model and the quality of post-trained models. By injecting special markers that signal when to prioritize creativity, the model can generate more diverse outputs. They also introduce a new metric based on graph edit distance to better capture narrative diversity, which traditional metrics might miss. In tests, CreativeInstruct not only matches but often exceeds the diversity of existing models while maintaining quality, and human evaluations show that people find its outputs more creative. This approach also benefits reinforcement learning applications, leading to improved performance in tasks like AMC and MATH compared to traditional post-trained models.

The method introduces a novel approach to balancing creativity and quality in LLM outputs.

The evaluation includes both quantitative metrics and human assessments, providing a solid basis for claims.