Binding Dynamics in Rotating Features
Sindy Löwe Francesco Locatello Max Welling
In human cognition, the binding problem describes the open question of how
the brain flexibly integrates diverse information into cohesive object
representations. Analogously, in machine learning, there is a pursuit for
models capable of strong generalization and reasoning by learning
object-centric representations in an unsupervised manner. Drawing from
neuroscientific theories, Rotating Features learn such representations by
introducing vector-valued features that encapsulate object characteristics in
their magnitudes and object affiliation in their orientations. The
"$\chi$-binding" mechanism, embedded in every layer of the architecture, has
been shown to be crucial, but remains poorly understood. In this paper, we
propose an alternative "cosine binding" mechanism, which explicitly computes
the alignment between features and adjusts weights accordingly, and we show
that it achieves equivalent performance. This allows us to draw direct
connections to self-attention and biological neural processes, and to shed
light on the fundamental dynamics for object-centric representations to emerge
in Rotating Features.