Hierarchical Continuous Diffusion Language Models

Imagine you're building a language model that needs to generate coherent text while also reasoning through complex structures, like solving a Sudoku puzzle. Current models often generate tokens one at a time, which can lead to inconsistencies because each token is chosen independently, breaking the connections that should exist between them. This issue, known as the independence of token sampling, can result in outputs that lack coherence or fail to satisfy global constraints. Continuous models try to address this by maintaining a shared state, but they struggle to connect that state back to valid token outputs until the very end of the process, which can limit their effectiveness in generating meaningful sequences. To tackle these challenges, the authors propose a new method that combines the strengths of both discrete and continuous models. By using a hierarchical approach, they create a continuous latent space that evolves as tokens are generated, allowing for a more integrated and coherent generation process. This method not only improves the accuracy of structured reasoning tasks like Sudoku and Countdown but also enhances the quality of language modeling on datasets like LM1B. For builders, this means a more robust framework for developing applications that require both reasoning and natural language generation, potentially leading to more reliable and contextually aware AI systems.

The approach introduces a novel coupling of discrete and continuous generation methods.

The evaluation on multiple tasks shows solid improvements over existing methods.