MultiSChuBERT: Effective Multimodal Fusion for Scholarly Document
  Quality Prediction
Gideon Maillette de Buy Wenniger Thomas van Dongen Lambert Schomaker
Automatic assessment of the quality of scholarly documents is a difficult
task with high potential impact. Multimodality, in particular the addition of
visual information next to text, has been shown to improve the performance on
scholarly document quality prediction (SDQP) tasks. We propose the multimodal
predictive model MultiSChuBERT. It combines a textual model based on chunking
full paper text and aggregating computed BERT chunk-encodings (SChuBERT), with
a visual model based on Inception V3.Our work contributes to the current
state-of-the-art in SDQP in three ways. First, we show that the method of
combining visual and textual embeddings can substantially influence the
results. Second, we demonstrate that gradual-unfreezing of the weights of the
visual sub-model, reduces its tendency to ovefit the data, improving results.
Third, we show the retained benefit of multimodality when replacing standard
BERT$_{\textrm{BASE}}$ embeddings with more recent state-of-the-art text
embedding models.
  Using BERT$_{\textrm{BASE}}$ embeddings, on the (log) number of citations
prediction task with the ACL-BiblioMetry dataset, our MultiSChuBERT
(text+visual) model obtains an $R^{2}$ score of 0.454 compared to 0.432 for the
SChuBERT (text only) model. Similar improvements are obtained on the PeerRead
accept/reject prediction task. In our experiments using SciBERT, scincl,
SPECTER and SPECTER2.0 embeddings, we show that each of these tailored
embeddings adds further improvements over the standard BERT$_{\textrm{BASE}}$
embeddings, with the SPECTER2.0 embeddings performing best.