Let's Do a Thought Experiment: Using Counterfactuals to Improve Moral
  Reasoning
Xiao Ma Swaroop Mishra Ahmad Beirami Alex Beutel Jilin Chen
Language models still struggle on moral reasoning, despite their impressive
performance in many other tasks. In particular, the Moral Scenarios task in
MMLU (Multi-task Language Understanding) is among the worst performing tasks
for many language models, including GPT-3. In this work, we propose a new
prompting framework, Thought Experiments, to teach language models to do better
moral reasoning using counterfactuals. Experiment results show that our
framework elicits counterfactual questions and answers from the model, which in
turn helps improve the accuracy on Moral Scenarios task by 9-16% compared to
other zero-shot baselines. Interestingly, unlike math reasoning tasks,
zero-shot Chain-of-Thought (CoT) reasoning doesn't work out of the box, and
even reduces accuracy by around 4% compared to direct zero-shot. We further
observed that with minimal human supervision in the form of 5 few-shot
examples, the accuracy of the task can be improved to as much as 80%.