Principled Architecture-aware Scaling of Hyperparameters
Wuyang Chen Junru Wu Zhangyang Wang Boris Hanin
Training a high-quality deep neural network requires choosing suitable
hyperparameters, which is a non-trivial and expensive process. Current works
try to automatically optimize or design principles of hyperparameters, such
that they can generalize to diverse unseen scenarios. However, most designs or
optimization methods are agnostic to the choice of network structures, and thus
largely ignore the impact of neural architectures on hyperparameters. In this
work, we precisely characterize the dependence of initializations and maximal
learning rates on the network architecture, which includes the network depth,
width, convolutional kernel size, and connectivity patterns. By pursuing every
parameter to be maximally updated with the same mean squared change in
pre-activations, we can generalize our initialization and learning rates across
MLPs (multi-layer perception) and CNNs (convolutional neural network) with
sophisticated graph topologies. We verify our principles with comprehensive
experiments. More importantly, our strategy further sheds light on advancing
current benchmarks for architecture design. A fair comparison of AutoML
algorithms requires accurate network rankings. However, we demonstrate that
network rankings can be easily changed by better training networks in
benchmarks with our architecture-aware learning rates and initialization.