On Influence Functions, Classification Influence, Relative Influence,
  Memorization and Generalization
Michael Kounavis Ousmane Dia Ilqar Ramazanli
Machine learning systems such as large scale recommendation systems or
natural language processing systems are usually trained on billions of training
points and are associated with hundreds of billions or trillions of parameters.
Improving the learning process in such a way that both the training load is
reduced and the model accuracy improved is highly desired. In this paper we
take a first step toward solving this problem, studying influence functions
from the perspective of simplifying the computations they involve. We discuss
assumptions, under which influence computations can be performed on
significantly fewer parameters. We also demonstrate that the sign of the
influence value can indicate whether a training point is to memorize, as
opposed to generalize upon. For this purpose we formally define what
memorization means for a training point, as opposed to generalization. We
conclude that influence functions can be made practical, even for large scale
machine learning systems, and that influence values can be taken into account
by algorithms that selectively remove training points, as part of the learning
process.