Tokenizer Transplantation: Mitigating Autoregressive Collapse in Edge-Efficient Bengali ASR
Sanjid Hasan, Md. Abdur Rahman
Read on arXiv →
Key claim
Vocabulary transplantation improves ASR for Bengali significantly.
In plain English
Lightweight speech recognition models struggle with languages like Bengali due to their reliance on English-centric tokenizers. This paper identifies that such tokenizers break down Bengali words, leading to poor performance. The authors propose a new method that replaces the existing vocabulary with one tailored for Bengali, which stabilizes the decoding process. This is important for builders looking to deploy effective speech recognition in diverse linguistic contexts without extensive retraining.
Introduces a novel vocabulary transplantation pipeline for ASR models.
Experimental results are solid with a competitive WER and RTF.
Deep reliability assessment
The methodology supports the claim that replacing the tokenizer with a native-script BanglaBERT tokenizer reduces token fertility and stabilizes decoding, but the claim of achieving a scalable blueprint for cross-script adaptation without resource-intensive pre-training may be overclaimed without broader validation.
Reproducibility
yes, the paper mentions that the full implementation details for the framework are available in their repository.
Key figure
The key architectural diagram likely illustrates the Tokenizer Transplantation pipeline, showing the process of replacing the English-centric tokenizer with a native-script BanglaBERT tokenizer.