Multi-scale Unified Network for Image Classification
Wenzhuo Liu Fei Zhu Cheng-Lin Liu
Convolutional Neural Networks (CNNs) have advanced significantly in visual
representation learning and recognition. However, they face notable challenges
in performance and computational efficiency when dealing with real-world,
multi-scale image inputs. Conventional methods rescale all input images into a
fixed size, wherein a larger fixed size favors performance but rescaling small
size images to a larger size incurs digitization noise and increased
computation cost. In this work, we carry out a comprehensive, layer-wise
investigation of CNN models in response to scale variation, based on Centered
Kernel Alignment (CKA) analysis. The observations reveal lower layers are more
sensitive to input image scale variations than high-level layers. Inspired by
this insight, we propose Multi-scale Unified Network (MUSN) consisting of
multi-scale subnets, a unified network, and scale-invariant constraint. Our
method divides the shallow layers into multi-scale subnets to enable feature
extraction from multi-scale inputs, and the low-level features are unified in
deep layers for extracting high-level semantic features. A scale-invariant
constraint is posed to maintain feature consistency across different scales.
Extensive experiments on ImageNet and other scale-diverse datasets, demonstrate
that MSUN achieves significant improvements in both model performance and
computational efficiency. Particularly, MSUN yields an accuracy increase up to
44.53% and diminishes FLOPs by 7.01-16.13% in multi-scale scenarios.