Evaluating Privacy Questions From Stack Overflow: Can ChatGPT Compete?
Zack Delile Sean Radel Joe Godinez Garrett Engstrom Theo Brucker Kenzie Young Sepideh Ghanavati
Stack Overflow and other similar forums are used commonly by developers to
seek answers for their software development as well as privacy-related
concerns. Recently, ChatGPT has been used as an alternative to generate code or
produce responses to developers' questions. In this paper, we aim to understand
developers' privacy challenges by evaluating the types of privacy-related
questions asked on Stack Overflow. We then conduct a comparative analysis
between the accepted responses given by Stack Overflow users and the responses
produced by ChatGPT for those extracted questions to identify if ChatGPT could
serve as a viable alternative. Our results show that most privacy-related
questions are related to choice/consent, aggregation, and identification.
Furthermore, our findings illustrate that ChatGPT generates similarly correct
responses for about 56% of questions, while for the rest of the responses, the
answers from Stack Overflow are slightly more accurate than ChatGPT.