Neural Textured Deformable Meshes for Robust Analysis-by-Synthesis
Angtian Wang Wufei Ma Alan Yuille Adam Kortylewski
Human vision demonstrates higher robustness than current AI algorithms under
out-of-distribution scenarios. It has been conjectured such robustness benefits
from performing analysis-by-synthesis. Our paper formulates triple vision tasks
in a consistent manner using approximate analysis-by-synthesis by
render-and-compare algorithms on neural features. In this work, we introduce
Neural Textured Deformable Meshes, which involve the object model with
deformable geometry that allows optimization on both camera parameters and
object geometries. The deformable mesh is parameterized as a neural field, and
covered by whole-surface neural texture maps, which are trained to have spatial
discriminability. During inference, we extract the feature map of the test
image and subsequently optimize the 3D pose and shape parameters of our model
using differentiable rendering to best reconstruct the target feature map. We
show that our analysis-by-synthesis is much more robust than conventional
neural networks when evaluated on real-world images and even in challenging
out-of-distribution scenarios, such as occlusion and domain shift. Our
algorithms are competitive with standard algorithms when tested on conventional
performance measures.