DeltaMerge-LowRes: Composing Language and Task Deltas for Low-Resource Adaptation
Son Ha Xuan, Xuan-Bach Le, Phat T. Tran-Truong
Read on arXiv →
Key claim
Cross-axis TIES improves multilingual task performance significantly.
In plain English
In low-resource NLP, adapting models to new languages and tasks typically requires expensive fine-tuning. Current methods often combine these adaptations, which can lead to inefficiencies. This paper proposes a new approach that separates language and task adaptations, merging them in a novel way that improves performance on several tasks. Builders might care because this could reduce the cost and complexity of model adaptation in multilingual settings.
Introduces a novel merging technique for language and task adaptation.
Demonstrates solid empirical results across multiple languages and tasks.
Deep reliability assessment
The methodology supports the claim that separate training of language and task deltas can be recombined effectively, but it does not establish superiority over traditional fine-tuning methods in all scenarios.
Reproducibility
Yes, the paper mentions the release of JSON traces and a claim ledger, but the specific code repository is not provided.
Key figure
Figure 1 illustrates the DeltaMerge-LowRes framework, showing how a base multilingual model branches into language and task deltas, which are then composed using various merging rules for evaluation.