Structured Pruning of Neural Networks for Constraints Learning
Matteo Cacciola Antonio Frangioni Andrea Lodi
In recent years, the integration of Machine Learning (ML) models with
Operation Research (OR) tools has gained popularity across diverse
applications, including cancer treatment, algorithmic configuration, and
chemical process optimization. In this domain, the combination of ML and OR
often relies on representing the ML model output using Mixed Integer
Programming (MIP) formulations. Numerous studies in the literature have
developed such formulations for many ML predictors, with a particular emphasis
on Artificial Neural Networks (ANNs) due to their significant interest in many
applications. However, ANNs frequently contain a large number of parameters,
resulting in MIP formulations that are impractical to solve, thereby impeding
scalability. In fact, the ML community has already introduced several
techniques to reduce the parameter count of ANNs without compromising their
performance, since the substantial size of modern ANNs presents challenges for
ML applications as it significantly impacts computational efforts during
training and necessitates significant memory resources for storage. In this
paper, we showcase the effectiveness of pruning, one of these techniques, when
applied to ANNs prior to their integration into MIPs. By pruning the ANN, we
achieve significant improvements in the speed of the solution process. We
discuss why pruning is more suitable in this context compared to other ML
compression techniques, and we identify the most appropriate pruning
strategies. To highlight the potential of this approach, we conduct experiments
using feed-forward neural networks with multiple layers to construct
adversarial examples. Our results demonstrate that pruning offers remarkable
reductions in solution times without hindering the quality of the final
decision, enabling the resolution of previously unsolvable instances.