A Differentiable Framework for End-to-End Learning of Hybrid Structured
  Compression
Moonjung Eo Suhyun Kang Wonjong Rhee
Filter pruning and low-rank decomposition are two of the foundational
techniques for structured compression. Although recent efforts have explored
hybrid approaches aiming to integrate the advantages of both techniques, their
performance gains have been modest at best. In this study, we develop a
\textit{Differentiable Framework~(DF)} that can express filter selection, rank
selection, and budget constraint into a single analytical formulation. Within
the framework, we introduce DML-S for filter selection, integrating scheduling
into existing mask learning techniques. Additionally, we present DTL-S for rank
selection, utilizing a singular value thresholding operator. The framework with
DML-S and DTL-S offers a hybrid structured compression methodology that
facilitates end-to-end learning through gradient-base optimization.
Experimental results demonstrate the efficacy of DF, surpassing
state-of-the-art structured compression methods. Our work establishes a robust
and versatile avenue for advancing structured compression techniques.