How does Architecture Influence the Base Capabilities of Pre-trained
  Language Models? A Case Study Based on FFN-Wider Transformer Models
Xin Lu Yanyan Zhao Bing Qin
Pre-trained language models have been proven to possess strong base
capabilities, which not only excel in in-distribution language modeling but
also show powerful abilities in out-of-distribution language modeling, transfer
learning and few-shot learning. Unlike existing work focusing on the influence
of scale on base capabilities, our work examines the influence of architecture
on those. Specifically, our concern is: How does architecture influence the
base capabilities of pre-trained language models? In this work, we attempt to
explain and reverse the decline in base capabilities caused by the architecture
of FFN-Wider Transformers, seeking to provide some insights. Through analysis,
we found the contribution ratio of Multi-Head Attention (a combination
function) to pre-trained language modeling is a key factor affecting base
capabilities. FFN-Wider Transformers reduce the contribution ratio of this
combination function, leading to a decline in base capabilities. We confirmed
this by experiments and proposed Combination Enhancement Architecture (CEA) to
address the decline in base capabilities of such models. Significantly, we
extended our explanation and CEA to Mixture of Experts (MoE) architecture
Transformers, which also alleviated their decline in base capabilities to some
extent, proving our work can offer useful guidance for architecture analysis,
architecture improvement and architecture design.