ConvergeFlow: Language Flow with Provable Convergence to Token Embeddings

Imagine you're building a language model that needs to generate text in a way that feels natural and coherent. Current models often rely on a method called cross-entropy to guide their predictions, but this can lead to issues where the generated outputs don't always correspond to valid words or phrases. This is problematic because it can result in nonsensical outputs, which is what's called a failure mode in language generation. The challenge is to create a model that can predict valid tokens directly without getting stuck in these pitfalls.

To address this, the authors propose a new method called ConvergeFlow, which operates in the embedding space of tokens. Instead of using cross-entropy, it constrains predictions to a specific area defined by valid token embeddings and trains the model using mean squared error. This approach ensures that even if there are errors in the predictions, the model can still converge to valid outputs. They also introduce various sampling techniques to balance the quality of generated text with the model's uncertainty. Compared to previous models, ConvergeFlow shows that it's possible to achieve competitive performance without the traditional reliance on cross-entropy, which could open new avenues for building more robust language models.

ConvergeFlow introduces a novel approach to token prediction without relying on cross-entropy supervision.

The experiments show competitive performance against existing models, but details on baselines could be clearer.