Comprehensive Solution Program Centric Pretraining for Table-and-Text
  Hybrid Numerical Reasoning
Qianying Liu Dongsheng Yang Wenjie Zhong Fei Cheng Sadao Kurohashi
Numerical reasoning over table-and-text hybrid passages, such as financial
reports, poses significant challenges and has numerous potential applications.
Noise and irrelevant variables in the model input have been a hindrance to its
performance. Additionally, coarse-grained supervision of the whole solution
program has impeded the model's ability to learn the underlying numerical
reasoning process. In this paper, we propose three pretraining tasks that
operate at both the whole program and sub-program level: Variable Integrity
Ranking, which guides the model to focus on useful variables; Variable Operator
Prediction, which decomposes the supervision into fine-grained single operator
prediction; and Variable Keyphrase Masking, which encourages the model to
identify key evidence that sub-programs are derived from. Experimental results
demonstrate the effectiveness of our proposed methods, surpassing
transformer-based model baselines.