Extracting Self-Consistent Causal Insights from Users Feedback with LLMs
  and In-context Learning
Sara Abdali Anjali Parikh Steve Lim Emre Kiciman
Microsoft Windows Feedback Hub is designed to receive customer feedback on a
wide variety of subjects including critical topics such as power and battery.
Feedback is one of the most effective ways to have a grasp of users' experience
with Windows and its ecosystem. However, the sheer volume of feedback received
by Feedback Hub makes it immensely challenging to diagnose the actual cause of
reported issues. To better understand and triage issues, we leverage Double
Machine Learning (DML) to associate users' feedback with telemetry signals. One
of the main challenges we face in the DML pipeline is the necessity of domain
knowledge for model design (e.g., causal graph), which sometimes is either not
available or hard to obtain. In this work, we take advantage of reasoning
capabilities in Large Language Models (LLMs) to generate a prior model that
which to some extent compensates for the lack of domain knowledge and could be
used as a heuristic for measuring feedback informativeness. Our LLM-based
approach is able to extract previously known issues, uncover new bugs, and
identify sequences of events that lead to a bug, while minimizing out-of-domain
outputs.