Set-based Neural Network Encoding
Bruno Andreis Soro Bedionita Sung Ju Hwang
We propose an approach to neural network weight encoding for generalization
performance prediction that utilizes set-to-set and set-to-vector functions to
efficiently encode neural network parameters. Our approach is capable of
encoding neural networks in a modelzoo of mixed architecture and different
parameter sizes as opposed to previous approaches that require custom encoding
models for different architectures. Furthermore, our \textbf{S}et-based
\textbf{N}eural network \textbf{E}ncoder (SNE) takes into consideration the
hierarchical computational structure of neural networks by utilizing a
layer-wise encoding scheme that culminates to encoding all layer-wise encodings
to obtain the neural network encoding vector. Additionally, we introduce a
\textit{pad-chunk-encode} pipeline to efficiently encode neural network layers
that is adjustable to computational and memory constraints. We also introduce
two new tasks for neural network generalization performance prediction:
cross-dataset and cross-architecture. In cross-dataset performance prediction,
we evaluate how well performance predictors generalize across modelzoos trained
on different datasets but of the same architecture. In cross-architecture
performance prediction, we evaluate how well generalization performance
predictors transfer to modelzoos of different architecture. Experimentally, we
show that SNE outperforms the relevant baselines on the cross-dataset task and
provide the first set of results on the cross-architecture task.