Method
World Action Models (WAMs) improve robot control by jointly predicting actions and future observations, but contact-rich manipulation requires physical information that vision alone cannot reliably capture. Recent tactile WAMs often rely on large pretrained generative backbones, which limit their inference efficiency and flexible deployment. To address this limitation, we present Agile-WAM, an agile tactile World Action Model for contact-rich robot control. Agile-WAM encodes visual and tactile observations into a shared latent that serves as the source of a direct vision-tactile-to-action flow-matching process, which can jointly generate latent representations of an action chunk and future visual/tactile latents. A key observation is that vision and tactile signals evolve at inherently different timescales: adjacent visual frames are often highly similar, whereas tactile signals can change abruptly upon contact. We therefore introduce multi-horizon multimodal prediction in Agile-WAM, which provides supervision for visual latent at a larger temporal offset while predicting the adjacent tactile latent to preserve fine-grained contact dynamics. Across nine simulated and five real-world contact-rich manipulation tasks, Agile-WAM demonstrates strong and robust performance, outperforming the strongest baseline in success rate while maintaining low inference latency. In particular, in five real-world experiments, Agile-WAM yields a relative gain of 29.4% in overall success rates while achieving inference latency of 11.9 ms. These results demonstrate that multimodal WAM can be achieved with an agile architecture suitable for precise and high-frequency robot control.
Agile-WAM effectively learns and executes corrective behaviors in real-world experiments. When the initial attempt fails to align the object with the target position, Agile-WAM can still recover under an obstructed visual view by gently maintaining contact with the surrounding surface and using tactile feedback to search for the correct insertion position. This recovery capability substantially improves the success rate over baselines, which often fail to correct the misalignment.
We evaluate Agile-WAM on nine simulated and five real-world contact-rich manipulation tasks. Across these tasks, we investigate its performance and computational efficiency.
Ablation studies of Agile-WAM. (a) Joint action–future modeling improves performance beyond the gain from increased latent capacity. (b) Jointly modeling both future visual and tactile observations provides the strongest performance under multimodal input. (c) Multi-horizon multimodal prediction better matches the different temporal characteristics of vision and tactile signals. (d) We vary the visual and tactile prediction loss weights to explore their respective influence on policy performance.
Baseline methods lack either tactile feedback or dynamic modeling, resulting in imprecise motions and an inability to recover once the object becomes misaligned.
@article{agile-wam,
title = {Agile-WAM: An Agile Tactile World Action Model for Contact-Rich Robot Control},
author = {Zhou, Hanchu and Lynch, Brendan and Goyal, Raman and Gao, Dechen and Kasap, Begum and Zhao, Boqi and Zhang, Junshan},
year = {20XX}
}