DynaLay: An Introspective Approach to Dynamic Layer Selection for Deep
  Networks
Mrinal Mathur Sergey Plis
Deep learning models have become increasingly computationally intensive,
requiring extensive computational resources and time for both training and
inference. A significant contributing factor to this challenge is the uniform
computational effort expended on each input example, regardless of its
complexity. We introduce \textbf{DynaLay}, an alternative architecture that
features a decision-making agent to adaptively select the most suitable layers
for processing each input, thereby endowing the model with a remarkable level
of introspection. DynaLay reevaluates more complex inputs during inference,
adjusting the computational effort to optimize both performance and efficiency.
The core of the system is a main model equipped with Fixed-Point Iterative
(FPI) layers, capable of accurately approximating complex functions, paired
with an agent that chooses these layers or a direct action based on the
introspection of the models inner state. The model invests more time in
processing harder examples, while minimal computation is required for easier
ones. This introspective approach is a step toward developing deep learning
models that "think" and "ponder", rather than "ballistically'' produce answers.
Our experiments demonstrate that DynaLay achieves accuracy comparable to
conventional deep models while significantly reducing computational demands.