ChatGPT and Bard Responses to Polarizing Questions
Abhay Goyal Muhammad Siddique Nimay Parekh Zach Schwitzky Clara Broekaert Connor Michelotti Allie Wong Lam Yin Cheung Robin O Hanlon Lam Yin Cheung Munmun De Choudhury Roy Ka-Wei Lee Navin Kumar
Recent developments in natural language processing have demonstrated the
potential of large language models (LLMs) to improve a range of educational and
learning outcomes. Of recent chatbots based on LLMs, ChatGPT and Bard have made
it clear that artificial intelligence (AI) technology will have significant
implications on the way we obtain and search for information. However, these
tools sometimes produce text that is convincing, but often incorrect, known as
hallucinations. As such, their use can distort scientific facts and spread
misinformation. To counter polarizing responses on these tools, it is critical
to provide an overview of such responses so stakeholders can determine which
topics tend to produce more contentious responses -- key to developing targeted
regulatory policy and interventions. In addition, there currently exists no
annotated dataset of ChatGPT and Bard responses around possibly polarizing
topics, central to the above aims. We address the indicated issues through the
following contribution: Focusing on highly polarizing topics in the US, we
created and described a dataset of ChatGPT and Bard responses. Broadly, our
results indicated a left-leaning bias for both ChatGPT and Bard, with Bard more
likely to provide responses around polarizing topics. Bard seemed to have fewer
guardrails around controversial topics, and appeared more willing to provide
comprehensive, and somewhat human-like responses. Bard may thus be more likely
abused by malicious actors. Stakeholders may utilize our findings to mitigate
misinformative and/or polarizing responses from LLMs