Improving classifier decision boundaries using nearest neighbors
Johannes Schneider
Neural networks are not learning optimal decision boundaries. We show that
decision boundaries are situated in areas of low training data density. They
are impacted by few training samples which can easily lead to overfitting. We
provide a simple algorithm performing a weighted average of the prediction of a
sample and its nearest neighbors' (computed in latent space) leading to a minor
favorable outcomes for a variety of important measures for neural networks. In
our evaluation, we employ various self-trained and pre-trained convolutional
neural networks to show that our approach improves (i) resistance to label
noise, (ii) robustness against adversarial attacks, (iii) classification
accuracy, and to some degree even (iv) interpretability. While improvements are
not necessarily large in all four areas, our approach is conceptually simple,
i.e., improvements come without any modification to network architecture,
training procedure or dataset. Furthermore, they are in stark contrast to prior
works that often require trade-offs among the four objectives or provide
valuable, but non-actionable insights.