Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach
  for Robust Manipulation
Marcel Torne Anthony Simeonov Zechu Li April Chan Tao Chen Abhishek Gupta Pulkit Agrawal
Imitation learning methods need significant human supervision to learn
policies robust to changes in object poses, physical disturbances, and visual
distractors. Reinforcement learning, on the other hand, can explore the
environment autonomously to learn robust behaviors but may require impractical
amounts of unsafe real-world data collection. To learn performant, robust
policies without the burden of unsafe real-world data collection or extensive
human supervision, we propose RialTo, a system for robustifying real-world
imitation learning policies via reinforcement learning in "digital twin"
simulation environments constructed on the fly from small amounts of real-world
data. To enable this real-to-sim-to-real pipeline, RialTo proposes an
easy-to-use interface for quickly scanning and constructing digital twins of
real-world environments. We also introduce a novel "inverse distillation"
procedure for bringing real-world demonstrations into simulated environments
for efficient fine-tuning, with minimal human intervention and engineering
required. We evaluate RialTo across a variety of robotic manipulation problems
in the real world, such as robustly stacking dishes on a rack, placing books on
a shelf, and six other tasks. RialTo increases (over 67%) in policy robustness
without requiring extensive human data collection. Project website and videos
at https://real-to-sim-to-real.github.io/RialTo/