Visual Imitation Learning with Calibrated Contrastive Representation
Yunke Wang Linwei Tao Bo Du Yutian Lin Chang Xu
Adversarial Imitation Learning (AIL) allows the agent to reproduce expert
behavior with low-dimensional states and actions. However, challenges arise in
handling visual states due to their less distinguishable representation
compared to low-dimensional proprioceptive features. While existing methods
resort to adopt complex network architectures or separate the process of
learning representation and decision-making, they overlook valuable intra-agent
information within demonstrations. To address this problem, this paper proposes
a simple and effective solution by incorporating calibrated contrastive
representative learning into visual AIL framework. Specifically, we present an
image encoder in visual AIL, utilizing a combination of unsupervised and
supervised contrastive learning to extract valuable features from visual
states. Based on the fact that the improved agent often produces demonstrations
of varying quality, we propose to calibrate the contrastive loss by treating
each agent demonstrations as a mixed sample. The incorporation of contrastive
learning can be jointly optimized with the AIL framework, without modifying the
architecture or incurring significant computational costs. Experimental results
on DMControl Suite demonstrate our proposed method is sample efficient and can
outperform other compared methods from different aspects.