Automatic Outlier Rectification via Optimal Transport
Jose Blanchet Jiajin Li Markus Pelger Greg Zanotti
In this paper, we propose a novel conceptual framework to detect outliers
using optimal transport with a concave cost function. Conventional outlier
detection approaches typically use a two-stage procedure: first, outliers are
detected and removed, and then estimation is performed on the cleaned data.
However, this approach does not inform outlier removal with the estimation
task, leaving room for improvement. To address this limitation, we propose an
automatic outlier rectification mechanism that integrates rectification and
estimation within a joint optimization framework. We take the first step to
utilize an optimal transport distance with a concave cost function to construct
a rectification set in the space of probability distributions. Then, we select
the best distribution within the rectification set to perform the estimation
task. Notably, the concave cost function we introduced in this paper is the key
to making our estimator effectively identify the outlier during the
optimization process. We discuss the fundamental differences between our
estimator and optimal transport-based distributionally robust optimization
estimator. finally, we demonstrate the effectiveness and superiority of our
approach over conventional approaches in extensive simulation and empirical
analyses for mean estimation, least absolute regression, and the fitting of
option implied volatility surfaces.