Puro-2B: Poor Lab's Qwen2-1.5B Trained on RTX 5090 within $5090

Imagine you're a researcher wanting to develop a powerful language model, but the costs of pretraining are astronomical, often exceeding a million dollars. This financial barrier limits access for many in academia and the open-source community. Current solutions, while helpful, still require significant resources, making it hard for smaller teams to compete or innovate. This situation is what's called a scalability issue, where the high costs prevent widespread experimentation and development in the field. To address this, the authors propose a new pretraining recipe that allows for training models at a fraction of the cost, using consumer-grade hardware and efficient training techniques. By leveraging low-precision training and optimizing the data and training process, they manage to train models that perform comparably to more expensive counterparts for under $7K. They also introduce a Puro Cost Scaling Law, which provides insights into how training costs relate to model performance, suggesting that even lower budgets can yield competitive results. This approach not only democratizes access to powerful language models but also encourages more experimentation and innovation in the field, as researchers can now train and fine-tune models without prohibitive costs.

The paper presents a cost-effective pretraining recipe that significantly lowers the barrier for training large language models.

The approach is backed by empirical results and a derived cost scaling law, enhancing its credibility.