Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case
  Study on Reddit
Parker Seegmiller Joseph Gatto Omar Sharif Madhusudan Basak Sarah Masud Preum
Large language models (LLMs) have been shown to be proficient in correctly
answering questions in the context of online discourse. However, the study of
using LLMs to model human-like answers to fact-driven social media questions is
still under-explored. In this work, we investigate how LLMs model the wide
variety of human answers to fact-driven questions posed on several
topic-specific Reddit communities, or subreddits. We collect and release a
dataset of 409 fact-driven questions and 7,534 diverse, human-rated answers
from 15 r/Ask{Topic} communities across 3 categories: profession, social
identity, and geographic location. We find that LLMs are considerably better at
modeling highly-rated human answers to such questions, as opposed to
poorly-rated human answers. We present several directions for future research
based on our initial findings.