Provably adaptive sampling with uniform and remasking discrete diffusion models

Imagine you're developing a generative model that needs to produce high-quality outputs quickly, like creating images or text. Traditional methods often rely on sequential updates, which can be slow and inefficient, especially as the complexity of the data increases. This inefficiency is particularly evident when the model's performance is heavily influenced by the dimensionality of the data, leading to what's known as linear scaling issues. When many variables are updated simultaneously, mistakes can compound, making it hard to correct errors during the generation process. This is what's called the 'sampling efficiency problem.'

To address this, the authors propose a new approach that allows for parallel updates during the sampling process, using a first-order sampler that can adaptively correct denoising mistakes. By focusing on the intrinsic structure of the target distribution rather than just the ambient dimension, they establish that fewer discretization steps are needed to achieve a desired level of accuracy. Their findings suggest that the complexity of sampling is more about the relationships within the data than the sheer number of dimensions. This shift in understanding could lead to more efficient generative models, making it easier for builders to create applications that require fast and reliable outputs without being bogged down by high dimensionality.

The paper introduces a new adaptive sampling method that improves efficiency in discrete diffusion models.

The results are supported by theoretical guarantees and numerical experiments, though the institution is unknown.