SocREval: Large Language Models with the Socratic Method for
  Reference-Free Reasoning Evaluation
Hangfeng He Hongming Zhang Dan Roth
To comprehensively assess the capacity of current models for complex
reasoning, it is crucial to assess their step-by-step reasoning in a scalable
manner. Established reference-based evaluation metrics rely on human-annotated
reasoning chains to assess the model-derived chains. However, such
``gold-standard'' human-written reasoning chains may not be unique and their
acquisition is often labor-intensive. Existing reference-free reasoning metrics
eliminate the need for human-crafted reasoning chains as references, but they
typically require fine-tuning on datasets with human-derived reasoning chains,
which complicates the process and raises concerns regarding generalizability
across diverse datasets. To address these challenges, we harness GPT-4 to
automatically evaluate reasoning chain quality, obviating the need for
human-crafted references. Leveraging the Socratic method, we devise tailored
prompts to enhance reference-free reasoning evaluation, which we term SocREval
(Socratic method for Reasoning Evaluation). Empirical results from four human
annotated datasets reveal that SocREval significantly improves GPT-4's
performance, surpassing existing reference-free and reference-based reasoning
evaluation metrics. Beyond its demonstrated efficacy, our proposed framework,
large language models (LLMs) with the Socratic method, proves to be both
cost-efficient and robust to prompt writing and example selection, as
substantiated by our in-depth analysis.