Telescopic Language Models
Zhilin Guo, Boqiao Zhang, Hakan Aktas, Kyle Fogarty, Nursena Koprucu Aslan, Wenzhao Li, Canberk Baykal, Albert Miao, Siyu Hong, Yixiao Liu, Adam Wu, Ashish Kumar Singh, Sakar Khattar, Chenliang Zhou, Weihao Xia, Cristina Nader Vasconcelos, Cengiz Oztireli
Read on arXiv →
Key claim
TLM enables efficient training across multiple model capacities.
In plain English
Imagine you're developing a language model that needs to perform well across different hardware setups, from powerful servers to mobile devices. Currently, if you want to optimize a model for each setup, you often have to train or compress a separate version for each one. This can lead to inefficiencies and wasted resources, especially when the model's performance varies significantly depending on the compute budget. This situation is what's called the 'fixed-exit' problem, where models are only trained to perform well at specific capacities, leaving them underperforming at others.
To address this, the authors propose a Telescopic Language Model (TLM), which is designed to be flexible across a continuum of capacities. The TLM is trained using a method called stochastic prefix supervision, where it learns to predict the next token while being supervised at various depths of its architecture. This means that at every training step, the model can operate effectively at multiple levels of complexity without needing to change its structure. The results show that TLM can reduce the overall cost of training while maintaining performance across different tasks, outperforming fixed-exit models in efficiency and perplexity. For builders, this means you can deploy a single model that adapts to your needs without the overhead of multiple training runs.
The approach introduces a novel training method that allows a single model to operate across multiple capacities without architectural changes.
The results are backed by empirical comparisons against fixed-exit models, though the baseline perplexity metrics could be more detailed.
Deep reliability assessment
The methodology supports the claim that a single TLM run can serve multiple compute budgets effectively, but it is unclear if this holds at larger scales beyond the tested 200M parameters.
Reproducibility
No open source code or dataset is mentioned as being available yet, but the paper states that the training and evaluation code will be released publicly upon publication.
Key figure
Figure 1 illustrates that a Telescopic Language Model (TLM) can serve as a valid language model at every depth, unlike vanilla LLMs and MLMS which only work at specific trained exits.