Bridging Offline-Online Evaluation with a Time-dependent and Popularity
  Bias-free Offline Metric for Recommenders
Petr Kasalický Rodrigo Alves Pavel Kordík
The evaluation of recommendation systems is a complex task. The offline and
online evaluation metrics for recommender systems are ambiguous in their true
objectives. The majority of recently published papers benchmark their methods
using ill-posed offline evaluation methodology that often fails to predict true
online performance. Because of this, the impact that academic research has on
the industry is reduced. The aim of our research is to investigate and compare
the online performance of offline evaluation metrics. We show that penalizing
popular items and considering the time of transactions during the evaluation
significantly improves our ability to choose the best recommendation model for
a live recommender system. Our results, averaged over five large-size
real-world live data procured from recommenders, aim to help the academic
community to understand better offline evaluation and optimization criteria
that are more relevant for real applications of recommender systems.