Stress Testing Chain-of-Thought Prompting for Large Language Models
Aayush Mishra Karan Thakkar
This report examines the effectiveness of Chain-of-Thought (CoT) prompting in
improving the multi-step reasoning abilities of large language models (LLMs).
Inspired by previous studies \cite{Min2022RethinkingWork}, we analyze the
impact of three types of CoT prompt perturbations, namely CoT order, CoT
values, and CoT operators on the performance of GPT-3 on various tasks. Our
findings show that incorrect CoT prompting leads to poor performance on
accuracy metrics. Correct values in the CoT is crucial for predicting correct
answers. Moreover, incorrect demonstrations, where the CoT operators or the CoT
order are wrong, do not affect the performance as drastically when compared to
the value based perturbations. This research deepens our understanding of CoT
prompting and opens some new questions regarding the capability of LLMs to
learn reasoning in context.