Can Large Language Models Infer Causal Relationships from Real-World Text?

The paper introduces ReCAST, the first benchmark to evaluate LLMs' ability to infer complex, realistic causal graphs from long-form, real-world academic texts, revealing that state-of-the-art models fail to exhibit robust causal reasoning under such conditions. Specifics below:

• Introduction of ReCAST: The first benchmark explicitly designed to evaluate LLMs’ ability to construct complex causal graphs from long-form, real-world texts (primarily economics literature), addressing gaps in prior synthetic and shallow-text benchmarks.

• Realistic Dataset Pipeline: A rigorous 3-stage pipeline (collection, annotation, post-processing) produces high-fidelity graph-text pairs, incorporating human expert annotation, LLM-aided normalization, and strict formatting standards to ensure semantic accuracy and reproducibility.

• Causal Graph Grounding from Narrative Text: Unlike prior work focusing on pairwise or sentence-level causality, ReCAST demands the extraction of multi-node causal networks from unstructured, naturally-written academic texts—a significantly harder and more authentic challenge.

• LLM-as-a-Judge Evaluation Framework: A novel automated evaluation method where LLMs assess generated graphs using semantic similarity, abstraction alignment, and relational correctness, allowing for nuanced grading beyond rigid structural or token-based comparisons.

• Degree of Confounding as Difficulty Metric: Introduction of a quantitative metric for implicitness, measuring how many graph nodes are not explicitly mentioned in the text, which serves as a predictor of LLM failure and a unique axis of analysis.

• Name-Assisted Graph Construction Ablation: A diagnostic setup where LLMs are given all ground-truth node names, isolating causal reasoning from entity extraction. Results show only marginal improvement, confirming causal inference as the primary performance bottleneck.

• Extensive Multi-Factor Error Analysis: Empirical study shows that LLMs fail due to inability to integrate dispersed information, abstract causal pathways, or avoid hallucinations—especially when causal relationships are implicit, multi-hop, or embedded in domain-specific jargon.