Critical Learning Periods Emerge Even in Deep Linear Networks
Michael Kleinman Alessandro Achille Stefano Soatto
Critical learning periods are periods early in development where temporary
sensory deficits can have a permanent effect on behavior and learned
representations. Despite the radical differences between biological and
artificial networks, critical learning periods have been empirically observed
in both systems. This suggests that critical periods may be fundamental to
learning and not an accident of biology. Yet, why exactly critical periods
emerge in deep networks is still an open question, and in particular it is
unclear whether the critical periods observed in both systems depend on
particular architectural or optimization details. To isolate the key underlying
factors, we focus on deep linear network models, and show that, surprisingly,
such networks also display much of the behavior seen in biology and artificial
networks, while being amenable to analytical treatment. We show that critical
periods depend on the depth of the model and structure of the data
distribution. We also show analytically and in simulations that the learning of
features is tied to competition between sources. Finally, we extend our
analysis to multi-task learning to show that pre-training on certain tasks can
damage the transfer performance on new tasks, and show how this depends on the
relationship between tasks and the duration of the pre-training stage. To the
best of our knowledge, our work provides the first analytically tractable model
that sheds light into why critical learning periods emerge in biological and
artificial networks.