Balanced and Deterministic Weight-sharing Helps Network Performance
Oscar Chang Hod Lipson
Weight-sharing plays a significant role in the success of many deep neural
networks, by increasing memory efficiency and incorporating useful inductive
priors about the problem into the network. But understanding how weight-sharing
can be used effectively in general is a topic that has not been studied
extensively. Chen et al. [2015] proposed HashedNets, which augments a
multi-layer perceptron with a hash table, as a method for neural network
compression. We generalize this method into a framework (ArbNets) that allows
for efficient arbitrary weight-sharing, and use it to study the role of
weight-sharing in neural networks. We show that common neural networks can be
expressed as ArbNets with different hash functions. We also present two novel
hash functions, the Dirichlet hash and the Neighborhood hash, and use them to
demonstrate experimentally that balanced and deterministic weight-sharing helps
with the performance of a neural network.