AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling

Imagine you're developing a language model that needs to generate coherent and contextually relevant text. Traditionally, these models rely on discrete tokens, which can limit their ability to capture the nuances of language. This approach often leads to issues like loss of fidelity in generated text or difficulties in modeling complex distributions, which is what's called token-level fidelity loss. Existing continuous models either compromise on the quality of the text representation or struggle with the generative process itself. This is where AURORA-LM comes in, offering a fresh perspective by maintaining a high-capacity text representation while allowing the diffusion model to learn its distribution directly. The model organizes text into a structured latent sequence and employs a diffusion transformer to generate text in a more coherent manner, addressing the shortcomings of previous methods. By focusing on a decodable text latent and refining the noise-level distribution, AURORA-LM enhances the quality of generated text significantly. Compared to prior work, it not only achieves superior performance on benchmarks like OpenWebText and XSum but also scales effectively, making it a compelling option for builders looking to create more sophisticated language models.

AURORA-LM introduces a new approach to continuous language modeling that separates text representation from distribution modeling.

The model achieves strong performance on established benchmarks, though details on baseline comparisons could be clearer.