QuadraNet: Improving High-Order Neural Interaction Efficiency with
  Hardware-Aware Quadratic Neural Networks
Chenhui Xu Fuxun Yu Zirui Xu Chenchen Liu Jinjun Xiong Xiang Chen
Recent progress in computer vision-oriented neural network designs is mostly
driven by capturing high-order neural interactions among inputs and features.
And there emerged a variety of approaches to accomplish this, such as
Transformers and its variants. However, these interactions generate a large
amount of intermediate state and/or strong data dependency, leading to
considerable memory consumption and computing cost, and therefore compromising
the overall runtime performance. To address this challenge, we rethink the
high-order interactive neural network design with a quadratic computing
approach. Specifically, we propose QuadraNet -- a comprehensive model design
methodology from neuron reconstruction to structural block and eventually to
the overall neural network implementation. Leveraging quadratic neurons'
intrinsic high-order advantages and dedicated computation optimization schemes,
QuadraNet could effectively achieve optimal cognition and computation
performance. Incorporating state-of-the-art hardware-aware neural architecture
search and system integration techniques, QuadraNet could also be well
generalized in different hardware constraint settings and deployment scenarios.
The experiment shows thatQuadraNet achieves up to 1.5$\times$ throughput, 30%
less memory footprint, and similar cognition performance, compared with the
state-of-the-art high-order approaches.