A Diachronic Perspective on User Trust in AI under Uncertainty
Shehzaad Dhuliawala Vilém Zouhar Mennatallah El-Assady Mrinmaya Sachan
In a human-AI collaboration, users build a mental model of the AI system
based on its reliability and how it presents its decision, e.g. its
presentation of system confidence and an explanation of the output. Modern NLP
systems are often uncalibrated, resulting in confidently incorrect predictions
that undermine user trust. In order to build trustworthy AI, we must understand
how user trust is developed and how it can be regained after potential
trust-eroding events. We study the evolution of user trust in response to these
trust-eroding events using a betting game. We find that even a few incorrect
instances with inaccurate confidence estimates damage user trust and
performance, with very slow recovery. We also show that this degradation in
trust reduces the success of human-AI collaboration and that different types of
miscalibration -- unconfidently correct and confidently incorrect -- have
different negative effects on user trust. Our findings highlight the importance
of calibration in user-facing AI applications and shed light on what aspects
help users decide whether to trust the AI system.