Systematic Architectural Design of Scale Transformed Attention Condenser
  DNNs via Multi-Scale Class Representational Response Similarity Analysis
Andre Hryniowski Alexander Wong
Self-attention mechanisms are commonly included in a convolutional neural
networks to achieve an improved efficiency performance balance. However, adding
self-attention mechanisms adds additional hyperparameters to tune for the
application at hand. In this work we propose a novel type of DNN analysis
called Multi-Scale Class Representational Response Similarity Analysis
(ClassRepSim) which can be used to identify specific design interventions that
lead to more efficient self-attention convolutional neural network
architectures. Using insights grained from ClassRepSim we propose the Spatial
Transformed Attention Condenser (STAC) module, a novel attention-condenser
based self-attention module. We show that adding STAC modules to ResNet style
architectures can result in up to a 1.6% increase in top-1 accuracy compared to
vanilla ResNet models and up to a 0.5% increase in top-1 accuracy compared to
SENet models on the ImageNet64x64 dataset, at the cost of up to 1.7% increase
in FLOPs and 2x the number of parameters. In addition, we demonstrate that
results from ClassRepSim analysis can be used to select an effective
parameterization of the STAC module resulting in competitive performance
compared to an extensive parameter search.