DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

Imagine you're developing a large language model (LLM) that needs to learn from vast amounts of text data. The challenge lies in how you prepare that data; many current methods apply a one-size-fits-all approach, treating every piece of data the same way. This can lead to inefficiencies and missed opportunities for improvement, especially when some data might need cleaning or specific handling while others do not. This is what's called a fixed processing strategy, and it often results in suboptimal performance because it doesn't adapt to the unique needs of each example.

To address this, the authors propose a framework called DataOrchestra, which customizes the data processing pipeline for each individual example. Instead of applying a uniform method, DataOrchestra decides on-the-fly whether to drop, leave untouched, or clean a chunk of data. If cleaning is needed, it selects appropriate operations, which can range from simple edits to complex LLM-based rewrites, generating specific instructions for each step. This tailored approach not only improves the model's performance across various benchmarks but also reduces unnecessary computational costs by skipping irrelevant processing steps. Compared to traditional methods, DataOrchestra offers a more nuanced and efficient way to prepare data for LLMs, making it a valuable tool for builders looking to enhance model training.

DataOrchestra introduces a flexible, example-specific data processing approach for LLMs.

The framework shows consistent performance improvements across multiple benchmarks, though specific baselines could be clearer.