Advancing Omnimodal Embodied Agents from Isolated Skills to Everyday Physical Autonomy
Junhao Shi, Zezheng Huai, Siyin Wang, Jia Chen, Yubang Wang, Zhaoye Fei, Hechang Chen, Jingjing Gong, Xipeng Qiu, Yu-Gang Jiang
Read on arXiv →
Key claim
OmniAct significantly improves robotic task success rates.
In plain English
Imagine you're trying to build a robot that can operate in unpredictable environments, like a home or a factory. You want it to not only perform tasks but also to adapt when things go wrong, like if it bumps into something or if a device it relies on stops working. Currently, many systems treat planning and execution as separate issues, which can lead to problems. For instance, if a robot's plan doesn't account for a sudden obstacle, it might just keep going without realizing it needs to change course. This is what's called open-loop execution, and it can lead to failures in real-world scenarios.
What this paper proposes is a new way to handle these challenges by creating a system that combines different types of tools and processes. Instead of relying on a single model that tries to do everything, it uses a hierarchical approach that separates planning, memory, and verification. This means that when the robot is executing a task, it can also check if things are going as planned and adjust if necessary. The OmniAct framework includes a multimodal planner that helps the robot choose the right actions, a memory system that keeps track of what’s important without getting overloaded, and a mechanism that allows it to interrupt its current task if something unexpected happens.
In practical terms, this means that OmniAct can handle a wide range of tasks more effectively than previous systems. It was tested on 40 real-world tasks with two different robotic platforms and showed consistent improvements in success rates, even as the complexity of the tasks increased. This is a big step forward for anyone looking to build autonomous systems that need to operate reliably in dynamic environments.
The paper introduces a hierarchical asynchronous architecture that integrates multiple domains, which is a significant extension of existing approaches.
The claims are supported by extensive real-world testing across multiple tasks and platforms, demonstrating solid performance improvements.
Deep reliability assessment
The methodology supports that a hierarchical planner-memory-verifier architecture can improve success and token efficiency on the authors' 40 real-world tasks across two robot platforms and four IoT devices. The broader claim of 'everyday physical autonomy' and mid-scale open models reaching proprietary-level performance is under-supported without more diverse homes, longer deployments, clearer quantitative tables, and stronger baselines.
Reproducibility
Code is open-sourced at the GitHub URL mentioned in the paper. The paper describes 40 real-world long-horizon tasks on two robotic platforms with four IoT devices, but the provided text does not indicate a released dataset or full task benchmark.
Key figure
Figure 1 contrasts two deployment challenges: coordinating IoT, APIs, manipulation, and navigation within one user command, and recovering autonomously when physical execution fails during long-horizon tasks.