HeroLT: Benchmarking Heterogeneous Long-Tailed Learning
Haohui Wang Weijie Guan Jianpeng Chen Zi Wang Dawei Zhou
Long-tailed data distributions are prevalent in a variety of domains,
including finance, e-commerce, biomedical science, and cyber security. In such
scenarios, the performance of machine learning models is often dominated by the
head categories, while the learning of tail categories is significantly
inadequate. Given abundant studies conducted to alleviate the issue, this work
aims to provide a systematic view of long-tailed learning with regard to three
pivotal angles: (A1) the characterization of data long-tailedness, (A2) the
data complexity of various domains, and (A3) the heterogeneity of emerging
tasks. To achieve this, we develop the most comprehensive (to the best of our
knowledge) long-tailed learning benchmark named HeroLT, which integrates 13
state-of-the-art algorithms and 6 evaluation metrics on 14 real-world benchmark
datasets across 4 tasks from 3 domains. HeroLT with novel angles and extensive
experiments (264 in total) enables researchers and practitioners to effectively
and fairly evaluate newly proposed methods compared with existing baselines on
varying types of datasets. Finally, we conclude by highlighting the significant
applications of long-tailed learning and identifying several promising future
directions. For accessibility and reproducibility, we open-source our benchmark
HeroLT and corresponding results at https://github.com/SSSKJ/HeroLT.