Eliciting Better Multilingual Structured Reasoning from LLMs through
  Code
Bryan Li Tamer Alkhouli Daniele Bonadiman Nikolaos Pappas Saab Mansour
Development of large language models (LLM) have shown progress on reasoning,
though studies have been limited to English or simple reasoning tasks. We thus
introduce a multilingual structured reasoning and explanation dataset, termed
xSTREET, that covers four tasks across six languages. xSTREET exposes a gap in
base LLM performance between English and non-English reasoning tasks. We then
propose two methods to remedy this gap, building on the insight that LLMs
trained on code are better reasoners. First, at training time, we augment a
code dataset with multi-lingual comments using machine translation while
keeping program code as-is. Second, at inference time, we bridge the gap
between training and inference by employing a prompt structure that
incorporates step-by-step code primitives to derive new facts and find a
solution. Our methods show improved multilingual performance on xSTREET, most
notably on the scientific commonsense reasoning subtask. Furthermore, the
models show no regression on non-reasoning tasks, thus showing our techniques
maintain general-purpose abilities.