MIMEx: Intrinsic Rewards from Masked Input Modeling
Toru Lin Allan Jabri
Exploring in environments with high-dimensional observations is hard. One
promising approach for exploration is to use intrinsic rewards, which often
boils down to estimating "novelty" of states, transitions, or trajectories with
deep networks. Prior works have shown that conditional prediction objectives
such as masked autoencoding can be seen as stochastic estimation of
pseudo-likelihood. We show how this perspective naturally leads to a unified
view on existing intrinsic reward approaches: they are special cases of
conditional prediction, where the estimation of novelty can be seen as
pseudo-likelihood estimation with different mask distributions. From this view,
we propose a general framework for deriving intrinsic rewards -- Masked Input
Modeling for Exploration (MIMEx) -- where the mask distribution can be flexibly
tuned to control the difficulty of the underlying conditional prediction task.
We demonstrate that MIMEx can achieve superior results when compared against
competitive baselines on a suite of challenging sparse-reward visuomotor tasks.