Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and
  unfairness in dyadic regression models
Jorge Paz-Ruza Amparo Alonso-Betanzos Bertha Guijarro-Berdiñas Brais Cancela Carlos Eiras-Franco
Dyadic regression models, which predict real-valued outcomes for pairs of
entities, are fundamental in many domains (e.g. predicting the rating of a user
to a product in Recommender Systems) and promising and under exploration in
many others (e.g. approximating the adequate dosage of a drug for a patient in
personalized pharmacology). In this work, we demonstrate that non-uniformity in
the observed value distributions of individual entities leads to severely
biased predictions in state-of-the-art models, skewing predictions towards the
average of observed past values for the entity and providing worse-than-random
predictive power in eccentric yet equally important cases. We show that the
usage of global error metrics like Root Mean Squared Error (RMSE) and Mean
Absolute Error (MAE) is insufficient to capture this phenomenon, which we name
eccentricity bias, and we introduce Eccentricity-Area Under the Curve (EAUC) as
a new complementary metric that can quantify it in all studied models and
datasets. We also prove the adequateness of EAUC by using naive de-biasing
corrections to demonstrate that a lower model bias correlates with a lower EAUC
and vice-versa. This work contributes a bias-aware evaluation of dyadic
regression models to avoid potential unfairness and risks in critical
real-world applications of such systems.