FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillation
Shuyi Zhang, Yunfan Lou, Hongyang Cheng, Yichen Guo, Chuyao Fu, Yaoxu Lyu, Xiaojie Zhang, Haoran Li, Pengwei Wang, Zhongyuan Wang, Shanghang Zhang
Read on arXiv →
Key claim
FORCE achieves 79% improvement in success rates without human intervention.
In plain English
Imagine you're trying to build a robot that can understand both visual and language inputs and act on them. The challenge is that these models often hit a ceiling in performance because they rely on imitation learning from sub-optimal data. When you try to improve them using Reinforcement Learning (RL), it can be really inefficient — you might end up wasting a lot of time and resources because the model struggles to learn effectively from its experiences. This is especially true when the model's initial learning phase is unstable, leading to what’s called catastrophic unlearning, and when it doesn't explore enough to find good actions, which forces you to step in and guide it manually. This is what's known as sample inefficiency in RL.
To tackle these issues, the authors propose a new approach called FORCE. The idea is to stabilize the learning process through a three-stage framework. First, it starts with a warm-up phase that helps the model adjust its learning based on high-quality data, which reduces the risk of the Q-function going off track. Then, during the main learning phase, this adjusted Q-function filters out low-value actions, ensuring that only the best options are considered for updates. This way, the model learns more effectively and efficiently.
In practical terms, FORCE has shown to significantly boost the success rates of these models by 79% and speed up training by over 30%. This means that if you're building autonomous robots, you can expect them to perform better and learn faster without needing constant human oversight.
FORCE introduces a new three-stage framework that effectively addresses key challenges in RL fine-tuning for VLA models.
The evaluation on various tasks shows significant improvements and is backed by solid experimental results.
Deep reliability assessment
The evidence supports FORCE improving sample efficiency and avoiding early RL fine-tuning collapse on a limited set of ManiSkill and Franka manipulation tasks, especially compared with BC/Cal-QL/ConRFT-style baselines. The broader claim that this is a general intervention-free route to autonomous VLA deployment feels overclaimed without more diverse robots, longer-horizon tasks, failure/safety analysis, and stronger reporting on variance and reproducibility.
Reproducibility
No public code repository, dataset release, or project URL is mentioned in the provided paper text. The paper names ManiSkill and a Franka Emika Panda real-robot setup, but the provided sections do not include enough release details to reproduce the full benchmark or real-world experiments directly.
Key figure
Figure 1 shows FORCE as a three-stage VLA fine-tuning pipeline: offline Cal-QL pretraining on demonstrations, value pre-calibration with mixed/static rollouts, and online fine-tuning where a calibrated critic filters expert and policy data for stable policy updates.