Empirical Risk Minimization for Losses without Variance
Guanhua Fang Ping Li Gennady Samorodnitsky
This paper considers an empirical risk minimization problem under
heavy-tailed settings, where data does not have finite variance, but only has
$p$-th moment with $p \in (1,2)$. Instead of using estimation procedure based
on truncated observed data, we choose the optimizer by minimizing the risk
value. Those risk values can be robustly estimated via using the remarkable
Catoni's method (Catoni, 2012). Thanks to the structure of Catoni-type
influence functions, we are able to establish excess risk upper bounds via
using generalized generic chaining methods. Moreover, we take computational
issues into consideration. We especially theoretically investigate two types of
optimization methods, robust gradient descent algorithm and empirical
risk-based methods. With an extensive numerical study, we find that the
optimizer based on empirical risks via Catoni-style estimation indeed shows
better performance than other baselines. It indicates that estimation directly
based on truncated data may lead to unsatisfactory results.