Early stopping by correlating online indicators in neural networks
Manuel Vilares Ferro Yerai Doval Mosquera Francisco J. Ribadas Pena Victor M. Darriba Bilbao
In order to minimize the generalization error in neural networks, a novel
technique to identify overfitting phenomena when training the learner is
formally introduced. This enables support of a reliable and trustworthy early
stopping condition, thus improving the predictive power of that type of
modeling. Our proposal exploits the correlation over time in a collection of
online indicators, namely characteristic functions for indicating if a set of
hypotheses are met, associated with a range of independent stopping conditions
built from a canary judgment to evaluate the presence of overfitting. That way,
we provide a formal basis for decision making in terms of interrupting the
learning process.
  As opposed to previous approaches focused on a single criterion, we take
advantage of subsidiarities between independent assessments, thus seeking both
a wider operating range and greater diagnostic reliability. With a view to
illustrating the effectiveness of the halting condition described, we choose to
work in the sphere of natural language processing, an operational continuum
increasingly based on machine learning. As a case study, we focus on parser
generation, one of the most demanding and complex tasks in the domain. The
selection of cross-validation as a canary function enables an actual comparison
with the most representative early stopping conditions based on overfitting
identification, pointing to a promising start toward an optimal bias and
variance control.