Attentive Multi-Layer Perceptron for Non-autoregressive Generation
Shuyang Jiang Jun Zhang Jiangtao Feng Lin Zheng Lingpeng Kong
Autoregressive~(AR) generation almost dominates sequence generation for its
efficacy. Recently, non-autoregressive~(NAR) generation gains increasing
popularity for its efficiency and growing efficacy. However, its efficiency is
still bottlenecked by quadratic complexity in sequence lengths, which is
prohibitive for scaling to long sequence generation and few works have been
done to mitigate this problem. In this paper, we propose a novel MLP variant,
\textbf{A}ttentive \textbf{M}ulti-\textbf{L}ayer \textbf{P}erceptron~(AMLP), to
produce a generation model with linear time and space complexity. Different
from classic MLP with static and learnable projection matrices, AMLP leverages
adaptive projections computed from inputs in an attentive mode. The
sample-aware adaptive projections enable communications among tokens in a
sequence, and model the measurement between the query and key space.
Furthermore, we marry AMLP with popular NAR models, deriving a highly efficient
NAR-AMLP architecture with linear time and space complexity. Empirical results
show that such marriage architecture surpasses competitive efficient NAR
models, by a significant margin on text-to-speech synthesis and machine
translation. We also test AMLP's self- and cross-attention ability separately
with extensive ablation experiments, and find them comparable or even superior
to the other efficient models. The efficiency analysis further shows that AMLP
extremely reduces the memory cost against vanilla non-autoregressive models for
long sequences.