Real In-Hand Manipulation Policy from Reinforcement Learning Expert

Project Description

The In-Hand Manipulator Policy is a robot policy developed at the Columbia ROAM lab on the Roam Hand 3 (RH3) meant to twist a lug nut in-hand. The hand takes in camera, proprioceptive, and tactile sensing inputs. An expert policy was first trained in simulation using reinforcement learning with discretized tactile sensor model. The expert policy was then distilled into an imitation learning action-chunking transformer policy using real-world rollout data collection.

Leveraging richer tactile data from real sensor signals, the distilled policy out-performed the expert RL policy, proving to be more robust and is able to twist the hexnut consistently for 5+ minutes withstanding external perturbations without failure.

Methods

Developed an in-hand manipulation policy for Columbia University’s ROAM Lab using the Roam Hand 3 (RH3) manipulation platform, integrating camera, proprioceptive, and tactile inputs to rotate a hex nut
Collected real-world rollouts from an existing reinforcement learning expert trained in domain-randomized simulation with a discretized tactile sensor model
Distilled the expert’s behavior into an Action-Chunking Transformer (ACT) policy through imitation learning on real-world rollout data including recovery behavior from perturbations
Incorporated richer, real-world tactile signals into the distilled policy, improving manipulation robustness beyond the simulation-trained RL expert
Achieved consistent in-hand nut rotation for 5+ minutes without failure, outperforming the expert policy in real-world execution

Media

Simulation Expert

Discretized Tactile Sensor Model

Real 7D Tactile Sensor