Dynamic Layer Tying for Parameter-Efficient Transformers
Tamir David Hay Lior Wolf
In the pursuit of reducing the number of trainable parameters in deep
transformer networks, we employ Reinforcement Learning to dynamically select
layers during training and tie them together. Every few iterations, the RL
agent is asked whether to train each layer $i$ independently or to copy the
weights of a previous layer $j<i$. This facilitates weight sharing, reduces the
number of trainable parameters, and also serves as an effective regularization
technique. Experimental evaluations validate that our model modestly
outperforms the baseline transformer model with regard to perplexity and
drastically reduces the number of trainable parameters. In particular, the
memory consumption during training is up to one order of magnitude less than
the conventional training method.