Evaluating ChatGPT as a Recommender System: A Rigorous Approach
Dario Di Palma Giovanni Maria Biancofiore Vito Walter Anelli Fedelucio Narducci Tommaso Di Noia Eugenio Di Sciascio
Recent popularity surrounds large AI language models due to their impressive
natural language capabilities. They contribute significantly to
language-related tasks, including prompt-based learning, making them valuable
for various specific tasks. This approach unlocks their full potential,
enhancing precision and generalization. Research communities are actively
exploring their applications, with ChatGPT receiving recognition. Despite
extensive research on large language models, their potential in recommendation
scenarios still needs to be explored. This study aims to fill this gap by
investigating ChatGPT's capabilities as a zero-shot recommender system. Our
goals include evaluating its ability to use user preferences for
recommendations, reordering existing recommendation lists, leveraging
information from similar users, and handling cold-start situations. We assess
ChatGPT's performance through comprehensive experiments using three datasets
(MovieLens Small, Last.FM, and Facebook Book). We compare ChatGPT's performance
against standard recommendation algorithms and other large language models,
such as GPT-3.5 and PaLM-2. To measure recommendation effectiveness, we employ
widely-used evaluation metrics like Mean Average Precision (MAP), Recall,
Precision, F1, normalized Discounted Cumulative Gain (nDCG), Item Coverage,
Expected Popularity Complement (EPC), Average Coverage of Long Tail (ACLT),
Average Recommendation Popularity (ARP), and Popularity-based Ranking-based
Equal Opportunity (PopREO). Through thoroughly exploring ChatGPT's abilities in
recommender systems, our study aims to contribute to the growing body of
research on the versatility and potential applications of large language
models. Our experiment code is available on the GitHub repository:
https://github.com/sisinflab/Recommender-ChatGPT