Emergent Equivariance in Deep Ensembles
Jan E. Gerken Pan Kessel
We demonstrate that deep ensembles are secretly equivariant models. More
precisely, we show that deep ensembles become equivariant for all inputs and at
all training times by simply using data augmentation. Crucially, equivariance
holds off-manifold and for any architecture in the infinite width limit. The
equivariance is emergent in the sense that predictions of individual ensemble
members are not equivariant but their collective prediction is. Neural tangent
kernel theory is used to derive this result and we verify our theoretical
insights using detailed numerical experiments.