Rolling the dice for better deep learning performance: A study of
  randomness techniques in deep neural networks
Mohammed Ghaith Altarabichi Sławomir Nowaczyk Sepideh Pashami Peyman Sheikholharam Mashhadi Julia Handl
This paper investigates how various randomization techniques impact Deep
Neural Networks (DNNs). Randomization, like weight noise and dropout, aids in
reducing overfitting and enhancing generalization, but their interactions are
poorly understood. The study categorizes randomness techniques into four types
and proposes new methods: adding noise to the loss function and random masking
of gradient updates. Using Particle Swarm Optimizer (PSO) for hyperparameter
optimization, it explores optimal configurations across MNIST, FASHION-MNIST,
CIFAR10, and CIFAR100 datasets. Over 30,000 configurations are evaluated,
revealing data augmentation and weight initialization randomness as main
performance contributors. Correlation analysis shows different optimizers
prefer distinct randomization types. The complete implementation and dataset
are available on GitHub.