Achieving Constraints in Neural Networks: A Stochastic Augmented
  Lagrangian Approach
Diogo Lavado Cláudia Soares Alessandra Micheletti
Regularizing Deep Neural Networks (DNNs) is essential for improving
generalizability and preventing overfitting. Fixed penalty methods, though
common, lack adaptability and suffer from hyperparameter sensitivity. In this
paper, we propose a novel approach to DNN regularization by framing the
training process as a constrained optimization problem. Where the data fidelity
term is the minimization objective and the regularization terms serve as
constraints. Then, we employ the Stochastic Augmented Lagrangian (SAL) method
to achieve a more flexible and efficient regularization mechanism. Our approach
extends beyond black-box regularization, demonstrating significant improvements
in white-box models, where weights are often subject to hard constraints to
ensure interpretability. Experimental results on image-based classification on
MNIST, CIFAR10, and CIFAR100 datasets validate the effectiveness of our
approach. SAL consistently achieves higher Accuracy while also achieving better
constraint satisfaction, thus showcasing its potential for optimizing DNNs
under constrained settings.