Tracking Object Positions in Reinforcement Learning: A Metric for
  Keypoint Detection (extended version)
Emma Cramer Jonas Reiher Sebastian Trimpe
Reinforcement learning (RL) for robot control typically requires a detailed
representation of the environment state, including information about
task-relevant objects not directly measurable. Keypoint detectors, such as
spatial autoencoders (SAEs), are a common approach to extracting a
low-dimensional representation from high-dimensional image data. SAEs aim at
spatial features such as object positions, which are often useful
representations in robotic RL. However, whether an SAE is actually able to
track objects in the scene and thus yields a spatial state representation well
suited for RL tasks has rarely been examined due to a lack of established
metrics. In this paper, we propose to assess the performance of an SAE instance
by measuring how well keypoints track ground truth objects in images. We
present a computationally lightweight metric and use it to evaluate common
baseline SAE architectures on image data from a simulated robot task. We find
that common SAEs differ substantially in their spatial extraction capability.
Furthermore, we validate that SAEs that perform well in our metric achieve
superior performance when used in downstream RL. Thus, our metric is an
effective and lightweight indicator of RL performance before executing
expensive RL training. Building on these insights, we identify three key
modifications of SAE architectures to improve tracking performance. We make our
code available at anonymous.4open.science/r/sae-rl.