Detection-based Intermediate Supervision for Visual Question Answering
Yuhang Liu Daowan Peng Wei Wei Yuanyuan Fu Wenfeng Xie Dangyang Chen
Recently, neural module networks (NMNs) have yielded ongoing success in
answering compositional visual questions, especially those involving multi-hop
visual and logical reasoning. NMNs decompose the complex question into several
sub-tasks using instance-modules from the reasoning paths of that question and
then exploit intermediate supervisions to guide answer prediction, thereby
improving inference interpretability. However, their performance may be
hindered due to sketchy modeling of intermediate supervisions. For instance,
(1) a prior assumption that each instance-module refers to only one grounded
object yet overlooks other potentially associated grounded objects, impeding
full cross-modal alignment learning; (2) IoU-based intermediate supervisions
may introduce noise signals as the bounding box overlap issue might guide the
model's focus towards irrelevant objects. To address these issues, a novel
method, \textbf{\underline{D}}etection-based \textbf{\underline{I}}ntermediate
\textbf{\underline{S}}upervision (DIS), is proposed, which adopts a generative
detection framework to facilitate multiple grounding supervisions via sequence
generation. As such, DIS offers more comprehensive and accurate intermediate
supervisions, thereby boosting answer prediction performance. Furthermore, by
considering intermediate results, DIS enhances the consistency in answering
compositional questions and their sub-questions.Extensive experiments
demonstrate the superiority of our proposed DIS, showcasing both improved
accuracy and state-of-the-art reasoning consistency compared to prior
approaches.