Text-driven Affordance Learning from Egocentric Vision
Tomoya Yoshida Shuhei Kurita Taichi Nishimura Shinsuke Mori
Visual affordance learning is a key component for robots to understand how to
interact with objects. Conventional approaches in this field rely on
pre-defined objects and actions, falling short of capturing diverse
interactions in realworld scenarios. The key idea of our approach is employing
textual instruction, targeting various affordances for a wide range of objects.
This approach covers both hand-object and tool-object interactions. We
introduce text-driven affordance learning, aiming to learn contact points and
manipulation trajectories from an egocentric view following textual
instruction. In our task, contact points are represented as heatmaps, and the
manipulation trajectory as sequences of coordinates that incorporate both
linear and rotational movements for various manipulations. However, when we
gather data for this task, manual annotations of these diverse interactions are
costly. To this end, we propose a pseudo dataset creation pipeline and build a
large pseudo-training dataset: TextAFF80K, consisting of over 80K instances of
the contact points, trajectories, images, and text tuples. We extend existing
referring expression comprehension models for our task, and experimental
results show that our approach robustly handles multiple affordances, serving
as a new standard for affordance learning in real-world scenarios.