Testing RadiX-Nets: Advances in Viable Sparse Topologies
Kevin Kwak Zack West Hayden Jananthan Jeremy Kepner
The exponential growth of data has sparked computational demands on ML
research and industry use. Sparsification of hyper-parametrized deep neural
networks (DNNs) creates simpler representations of complex data. Past research
has shown that some sparse networks achieve similar performance as dense ones,
reducing runtime and storage. RadiX-Nets, a subgroup of sparse DNNs, maintain
uniformity which counteracts their lack of neural connections. Generation,
independent of a dense network, yields faster asymptotic training and removes
the need for costly pruning. However, little work has been done on RadiX-Nets,
making testing challenging. This paper presents a testing suite for RadiX-Nets
in TensorFlow. We test RadiX-Net performance to streamline processing in
scalable models, revealing relationships between network topology,
initialization, and training behavior. We also encounter "strange models" that
train inconsistently and to lower accuracy while models of similar sparsity
train well.