ViTacFormer tactile architecture hits 40% success on deformable manipulation

ROBOTIS researchers tested three imitation learning approaches on a contact-rich bimanual task using their AI Worker platform with dual HX5-D20 hands. The baseline ACT policy using only RGB and joint states failed completely at separating nested paper cups. Adding raw tactile pressure arrays as MLP-projected tokens improved stability but only reached 10% success. The ViTacFormer architecture, which predicts future tactile states before decoding actions, reached 40% success by enabling the robot to anticipate grip force changes needed to peel cups apart.

How ViTacFormer structures tactile prediction

• Task: Bimanual unstacking and restacking of nested paper cups across 54-DoF action space
• Hardware: ROBOTIS AI Worker with dual 20-DoF HX5-D20 hands, each fingertip equipped with tactile pressure array
• Baseline failure: Vision-only ACT (0%) could not regulate contact force, either knocking cups over or failing to separate them
• Simple fusion limit: Tactile ACT (10%) added MLP-projected pressure arrays as tokens; improved grasp stability but still lifted both cups together
• ViTacFormer approach: Two-stage architecture predicts future tactile states from visual + proprioceptive + tactile history, then conditions action decoder on predicted touch tokens
• Result: 40% success rate, with the robot able to peel off top cup, stack it, and complete remaining stack

Why it matters

The result suggests that simply adding tactile sensors to robot hardware without architectural redesign yields limited gains for deformable manipulation. The open-source release of code and documentation may help ROS 2 researchers working on physical AI to move beyond vision-dominated policies.