PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

Imagine you're building a system that needs to handle complex reasoning tasks without breaking the bank. Large language models (LLMs) are great at this but can be prohibitively expensive to run at scale, while smaller models (SLMs) are cheaper but often struggle with difficult problems. This leads to a dilemma: how do you leverage the strengths of both without incurring high costs? The common approach is to use LLMs for everything, which can lead to inefficiencies and high operational costs, especially when the SLM could handle simpler tasks on its own. This is what's called over-reliance on LLMs, where the system doesn't optimize for cost-effectiveness in its reasoning process.

To address this, the authors propose PyroDash, a framework that allows SLMs to decide when to call on an LLM for help. The SLM emits a control token to signal when it needs assistance, and a Collaborate Engine manages the handoff to the LLM. This setup is efficient because it doesn't require retraining the LLM or using a separate routing mechanism. PyroDash trains the SLM in stages to optimize for both accuracy and cost, balancing the two through a reward system. The results show that PyroDash can significantly reduce costs while improving accuracy on reasoning tasks, making it a practical solution for those looking to deploy AI systems that need to be both effective and economical.

The approach introduces a novel cost-aware framework for SLM-LLM collaboration.

Results are based on multiple benchmarks, showing consistent improvements over baselines.