Thinking Like an Expert:Multimodal Hypergraph-of-Thought (HoT) Reasoning
  to boost Foundation Modals
Fanglong Yao Changyuan Tian Jintao Liu Zequn Zhang Qing Liu Li Jin Shuchao Li Xiaoyu Li Xian Sun
Reasoning ability is one of the most crucial capabilities of a foundation
model, signifying its capacity to address complex reasoning tasks.
Chain-of-Thought (CoT) technique is widely regarded as one of the effective
methods for enhancing the reasoning ability of foundation models and has
garnered significant attention. However, the reasoning process of CoT is
linear, step-by-step, similar to personal logical reasoning, suitable for
solving general and slightly complicated problems. On the contrary, the
thinking pattern of an expert owns two prominent characteristics that cannot be
handled appropriately in CoT, i.e., high-order multi-hop reasoning and
multimodal comparative judgement. Therefore, the core motivation of this paper
is transcending CoT to construct a reasoning paradigm that can think like an
expert. The hyperedge of a hypergraph could connect various vertices, making it
naturally suitable for modelling high-order relationships. Inspired by this,
this paper innovatively proposes a multimodal Hypergraph-of-Thought (HoT)
reasoning paradigm, which enables the foundation models to possess the
expert-level ability of high-order multi-hop reasoning and multimodal
comparative judgement. Specifically, a textual hypergraph-of-thought is
constructed utilizing triple as the primary thought to model higher-order
relationships, and a hyperedge-of-thought is generated through multi-hop
walking paths to achieve multi-hop inference. Furthermore, we devise a visual
hypergraph-of-thought to interact with the textual hypergraph-of-thought via
Cross-modal Co-Attention Graph Learning for multimodal comparative
verification. Experimentations on the ScienceQA benchmark demonstrate the
proposed HoT-based T5 outperforms CoT-based GPT3.5 and chatGPT, which is on par
with CoT-based GPT4 with a lower model size.