Modeling of learning curves with applications to pos tagging
Manuel Vilares Ferro Victor M. Darriba Bilbao Francisco J. Ribadas Pena
An algorithm to estimate the evolution of learning curves on the whole of a
training data base, based on the results obtained from a portion and using a
functional strategy, is introduced. We approximate iteratively the sought value
at the desired time, independently of the learning technique used and once a
point in the process, called prediction level, has been passed. The proposal
proves to be formally correct with respect to our working hypotheses and
includes a reliable proximity condition. This allows the user to fix a
convergence threshold with respect to the accuracy finally achievable, which
extends the concept of stopping criterion and seems to be effective even in the
presence of distorting observations.
  Our aim is to evaluate the training effort, supporting decision making in
order to reduce the need for both human and computational resources during the
learning process. The proposal is of interest in at least three operational
procedures. The first is the anticipation of accuracy gain, with the purpose of
measuring how much work is needed to achieve a certain degree of performance.
The second relates the comparison of efficiency between systems at training
time, with the objective of completing this task only for the one that best
suits our requirements. The prediction of accuracy is also a valuable item of
information for customizing systems, since we can estimate in advance the
impact of settings on both the performance and the development costs. Using the
generation of part-of-speech taggers as an example application, the
experimental results are consistent with our expectations.