LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Training
Shun Lei, Huaicheng Zhang, Dapeng Wu, Yaoxun Xu, Lishi Zuo, Wei Tan, Hangting Chen, Guangzheng Li, Jianwei Yu, Zhiyong Wu, Dong Yu
Read on arXiv →
Key claim
LeVo 2 outperforms existing systems in song generation quality.
LeVo 2 significantly improves full-length song generation by addressing the trade-off between vocal-instrument coordination and track-specific details. It achieves this through a hierarchical modeling approach that enhances both musicality and controllability, outperforming existing systems in subjective evaluations.
In plain English
Imagine you're trying to create a full-length song that sounds coherent and musical, while also ensuring that the vocals and instruments work well together. This is a tough challenge because existing systems often struggle with either keeping the overall sound cohesive or providing detailed sound quality for each track. When they focus on one aspect, they tend to lose out on the other, leading to songs that might sound good in parts but not as a whole. This is what's called a structural trade-off in song generation.
To tackle this, LeVo 2 introduces a clever way to manage these competing needs. It starts by predicting the overall structure of the song using a mixed-token approach, which helps in planning the song's semantics. Then, it refines the details for vocals and instruments in parallel, ensuring that both elements are well-coordinated. Additionally, it uses a diffusion-based Music Codec to reconstruct the final audio, which helps in maintaining high sound quality.
What sets LeVo 2 apart from previous methods is its aesthetics-guided training schedule. This means that during training, it uses an automated system to evaluate the musicality of the songs, which helps in aligning the model's outputs with what sounds good. The results show that LeVo 2 not only improves the quality of song generation but also enhances controllability and musicality, outperforming many existing systems. For anyone building music generation tools, this approach offers a more balanced and effective way to create songs that sound great from start to finish.
LeVo 2 introduces a hybrid approach that effectively balances semantic planning and track-specific refinement, advancing the state of the art in song generation.
The paper provides strong experimental validation through expert listening tests and objective evaluations against open-source baselines.
Deep reliability assessment
The methodology supports the claim that the hierarchical mixed-token-to-dual-track architecture and progressive DPO schedule improve their own LeVo 2 variants under MOS, Gemini-based prompt metrics, ASR, and PER evaluations. The stronger claim that it approaches commercial systems is harder to assess from the provided excerpt because the commercial comparison details, test set construction, rater protocol, and potential training-data overlap are not visible.
Reproducibility
No open-source code, model weights, or dataset release is mentioned in the provided abstract, introduction, results, limitations, or conclusion excerpts. The paper describes large-scale proprietary-style training data, automated aesthetic labeling, expert listening tests, and Gemini-based evaluation, but not enough detail to fully reproduce the system.
Discussion questions
- 1.LeVo 2 uses mixed tokens as a semantic planner and then predicts vocal/accompaniment tokens for refinement. If you were building a music product, would you prefer this hierarchy over a simpler dual-track-only model even if it adds architectural complexity and more training stages?
- 2.The paper argues that offline DPO improves controllability while semi-online DPO improves musicality, and that mixing objectives causes gradient conflict. Has anyone seen this kind of controllability-vs-aesthetic-quality tradeoff in generation systems, and would you solve it with staged training or with better reward modeling?
- 3.A lot of the alignment pipeline depends on an automated music aesthetic evaluation framework that assigns musicality tiers before preference alignment. Do you trust aesthetic labels produced automatically enough to bake them into pretraining, or would this risk amplifying the evaluator's taste biases?
- 4.The ablations show that removing the delay pattern causes PER to jump from 8.55% to 47.10% and OVL to drop from 5.48 to 4.65. Does that make the delay mechanism the real core contribution, or is it just patching a weakness created by the hierarchical token design?
- 5.The paper reports MOS, Gemini evaluation, ASR instrument score, and PER, but the most important user-facing metric may be whether people would actually reuse, edit, or publish the generated song. What evaluation would you add first if you were deciding whether this is product-ready?
Key figure
Figure 1 shows LeVo 2 as a hybrid LLM-diffusion pipeline where LeLM predicts mixed semantic tokens and dual-track vocal/accompaniment tokens, a diffusion Music Codec reconstructs audio, and training proceeds through aesthetic-conditioned pretraining, progressive post-training, and modular track-specific extension.