PartialFormer: Modeling Part Instead of Whole
Tong Zheng Bei Li Huiwen Bao Weiqiao Shan Tong Xiao Jingbo Zhu
The design choices in Transformer feed-forward neural networks have resulted
in significant computational and parameter overhead. In this work, we emphasize
the importance of hidden dimension in designing lightweight FFNs, a factor
often overlooked in previous architectures. Guided by this principle, we
introduce PartialFormer, a parameter-efficient Transformer architecture
utilizing multiple smaller FFNs to reduce parameters and computation while
maintaining essential hidden dimensions. These smaller FFNs are integrated into
a multi-head attention system to enable effective collaboration. We also
propose a tailored head scaling strategy to enhance PartialFormer's
capabilities. Furthermore, we present a residual-like attention calculation to
improve depth scaling within PartialFormer. Extensive experiments on 9
translation tasks and 1 abstractive summarization task validate the
effectiveness of our PartialFormer approach. Our code would be available at:
\url{https://github.com/zhengkid/PartialFormer}.