Chain-of-Action: Faithful and Multimodal Question Answering through
  Large Language Models
Zhenyu Pan Haozheng Luo Manling Li Han Liu
We present a Chain-of-Action (CoA) framework for multimodal and
retrieval-augmented Question-Answering (QA). Compared to the literature, CoA
overcomes two major challenges of current QA applications: (i) unfaithful
hallucination that is inconsistent with real-time or domain facts and (ii) weak
reasoning performance over compositional information. Our key contribution is a
novel reasoning-retrieval mechanism that decomposes a complex question into a
reasoning chain via systematic prompting and pre-designed actions.
Methodologically, we propose three types of domain-adaptable `Plug-and-Play'
actions for retrieving real-time information from heterogeneous sources. We
also propose a multi-reference faith score (MRFS) to verify and resolve
conflicts in the answers. Empirically, we exploit both public benchmarks and a
Web3 case study to demonstrate the capability of CoA over other methods.