ChatGPT Prompting Cannot Estimate Predictive Uncertainty in
  High-Resource Languages
Martino Pelucchi Matias Valdenegro-Toro
ChatGPT took the world by storm for its impressive abilities. Due to its
release without documentation, scientists immediately attempted to identify its
limits, mainly through its performance in natural language processing (NLP)
tasks. This paper aims to join the growing literature regarding ChatGPT's
abilities by focusing on its performance in high-resource languages and on its
capacity to predict its answers' accuracy by giving a confidence level. The
analysis of high-resource languages is of interest as studies have shown that
low-resource languages perform worse than English in NLP tasks, but no study so
far has analysed whether high-resource languages perform as well as English.
The analysis of ChatGPT's confidence calibration has not been carried out
before either and is critical to learn about ChatGPT's trustworthiness. In
order to study these two aspects, five high-resource languages and two NLP
tasks were chosen. ChatGPT was asked to perform both tasks in the five
languages and to give a numerical confidence value for each answer. The results
show that all the selected high-resource languages perform similarly and that
ChatGPT does not have a good confidence calibration, often being overconfident
and never giving low confidence values.