Early Period of Training Impacts Out-of-Distribution Generalization
Chen Cecilia Liu Iryna Gurevych
Prior research has found that differences in the early period of neural
network training significantly impact the performance of in-distribution (ID)
tasks. However, neural networks are often sensitive to out-of-distribution
(OOD) data, making them less reliable in downstream applications. Yet, the
impact of the early training period on OOD generalization remains understudied
due to its complexity and lack of effective analytical methodologies. In this
work, we investigate the relationship between learning dynamics and OOD
generalization during the early period of neural network training. We utilize
the trace of Fisher Information and sharpness, with a focus on gradual
unfreezing (i.e. progressively unfreezing parameters during training) as the
methodology for investigation. Through a series of empirical experiments, we
show that 1) selecting the number of trainable parameters at different times
during training, i.e. realized by gradual unfreezing -- has a minuscule impact
on ID results, but greatly affects the generalization to OOD data; 2) the
absolute values of sharpness and trace of Fisher Information at the initial
period of training are not indicative for OOD generalization, but the relative
values could be; 3) the trace of Fisher Information and sharpness may be used
as indicators for the removal of interventions during early period of training
for better OOD generalization.