REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing

Imagine you're building a speech recognition system that needs to transcribe audio accurately, including when there are long pauses or non-speech segments. The challenge is that when the system generates timestamps for the transcription, these can drift over time, especially during these non-speech periods. This drift means that while the words might still make sense, the timing can be completely off, leading to confusion in applications that rely on precise timing, like subtitles or voice commands. This issue is known as timestamp drift, and it can be particularly problematic in long audio segments where the system struggles to maintain accurate timing without additional context.

Currently, many systems try to correct this drift by fine-tuning the model with additional data, but this can lead to a forgetting problem where the model loses its ability to perform well on other tasks. This is where the new approach comes in. The authors propose a method called REDDIT, which stands for Replay-based Distribution Editing. This method allows the model to correct its timestamps without losing its original capabilities. It does this by editing the timestamps based on the model's own previous outputs while ensuring that it maintains its performance on non-timestamp tasks.

The results are quite striking: using this new framework, the authors were able to improve the accuracy of timestamp alignment significantly while only updating a small fraction of the model's parameters. This means that for builders working on ASR systems, they can achieve much better performance in terms of timing without the risk of degrading the overall system's capabilities.

The paper introduces a new framework for correcting timestamp drift in ASR systems, which is a significant advancement in the field.

The claims are well-supported by extensive benchmarks and improvements in performance metrics.