On Regularization and Inference with Label Constraints
Kaifu Wang Hangfeng He Tin D. Nguyen Piyush Kumar Dan Roth
Prior knowledge and symbolic rules in machine learning are often expressed in
the form of label constraints, especially in structured prediction problems. In
this work, we compare two common strategies for encoding label constraints in a
machine learning pipeline, regularization with constraints and constrained
inference, by quantifying their impact on model performance. For
regularization, we show that it narrows the generalization gap by precluding
models that are inconsistent with the constraints. However, its preference for
small violations introduces a bias toward a suboptimal model. For constrained
inference, we show that it reduces the population risk by correcting a model's
violation, and hence turns the violation into an advantage. Given these
differences, we further explore the use of two approaches together and propose
conditions for constrained inference to compensate for the bias introduced by
regularization, aiming to improve both the model complexity and optimal risk.