Chain-of-Thought Unfaithfulness as Disguised Accuracy
Oliver Bentham Nathan Stringham Ana Marasović
Understanding the extent to which Chain-of-Thought (CoT) generations align
with a large language model's (LLM) internal computations is critical for
deciding whether to trust an LLM's output. As a proxy for CoT faithfulness,
arXiv:2307.13702 propose a metric that measures a model's dependence on its CoT
for producing an answer. Within a single family of proprietary models, they
find that LLMs exhibit a scaling-then-inverse-scaling relationship between
model size and their measure of faithfulness, and that a 13 billion parameter
model exhibits increased faithfulness compared to models ranging from 810
million to 175 billion parameters in size. We evaluate whether these results
generalize as a property of all LLMs. We replicate their experimental setup
with three different families of models and, under specific conditions,
successfully reproduce the scaling trends for CoT faithfulness they report.
However, we discover that simply changing the order of answer choices in the
prompt can reduce the metric by 73 percentage points. The faithfulness metric
is also highly correlated ($R^2$ = 0.91) with accuracy, raising doubts about
its validity as a construct for evaluating faithfulness.