Adaptive recurrent vision performs zero-shot computation scaling to
  unseen difficulty levels
Vijay Veerabadran Srinivas Ravishankar Yuan Tang Ritik Raina Virginia R. de Sa
Humans solving algorithmic (or) reasoning problems typically exhibit solution
times that grow as a function of problem difficulty. Adaptive recurrent neural
networks have been shown to exhibit this property for various
language-processing tasks. However, little work has been performed to assess
whether such adaptive computation can also enable vision models to extrapolate
solutions beyond their training distribution's difficulty level, with prior
work focusing on very simple tasks. In this study, we investigate a critical
functional role of such adaptive processing using recurrent neural networks: to
dynamically scale computational resources conditional on input requirements
that allow for zero-shot generalization to novel difficulty levels not seen
during training using two challenging visual reasoning tasks: PathFinder and
Mazes. We combine convolutional recurrent neural networks (ConvRNNs) with a
learnable halting mechanism based on Graves (2016). We explore various
implementations of such adaptive ConvRNNs (AdRNNs) ranging from tying weights
across layers to more sophisticated biologically inspired recurrent networks
that possess lateral connections and gating. We show that 1) AdRNNs learn to
dynamically halt processing early (or late) to solve easier (or harder)
problems, 2) these RNNs zero-shot generalize to more difficult problem settings
not shown during training by dynamically increasing the number of recurrent
iterations at test time. Our study provides modeling evidence supporting the
hypothesis that recurrent processing enables the functional advantage of
adaptively allocating compute resources conditional on input requirements and
hence allowing generalization to harder difficulty levels of a visual reasoning
problem without training.