MinT: Boosting Generalization in Mathematical Reasoning via Multi-View
  Fine-Tuning
Zhenwen Liang Dian Yu Xiaoman Pan Wenlin Yao Qingkai Zeng Xiangliang Zhang Dong Yu
Reasoning in mathematical domains remains a significant challenge for
relatively small language models (LMs). Many current methods focus on
specializing LMs in mathematical reasoning and rely heavily on knowledge
distillation from powerful but inefficient large LMs (LLMs). In this work, we
explore a new direction that avoids over-reliance on LLM teachers, introducing
a multi-view fine-tuning method that efficiently exploits existing mathematical
problem datasets with diverse annotation styles. Our approach uniquely
considers the various annotation formats as different "views" and leverages
them in training the model. By postpending distinct instructions to input
questions, models can learn to generate solutions in diverse formats in a
flexible manner. Experimental results show that our strategy enables a LLaMA-7B
model to outperform prior approaches that utilize knowledge distillation, as
well as carefully established baselines. Additionally, the proposed method
grants the models promising generalization ability across various views and
datasets, and the capability to learn from inaccurate or incomplete noisy data.
We hope our multi-view training paradigm could inspire future studies in other
machine reasoning domains.