Sparse Double Descent in Vision Transformers: real or phantom threat?
Victor Quétu Marta Milovanovic Enzo Tartaglione
Vision transformers (ViT) have been of broad interest in recent theoretical
and empirical works. They are state-of-the-art thanks to their attention-based
approach, which boosts the identification of key features and patterns within
images thanks to the capability of avoiding inductive bias, resulting in highly
accurate image analysis. Meanwhile, neoteric studies have reported a ``sparse
double descent'' phenomenon that can occur in modern deep-learning models,
where extremely over-parametrized models can generalize well. This raises
practical questions about the optimal size of the model and the quest over
finding the best trade-off between sparsity and performance is launched: are
Vision Transformers also prone to sparse double descent? Can we find a way to
avoid such a phenomenon? Our work tackles the occurrence of sparse double
descent on ViTs. Despite some works that have shown that traditional
architectures, like Resnet, are condemned to the sparse double descent
phenomenon, for ViTs we observe that an optimally-tuned $\ell_2$ regularization
relieves such a phenomenon. However, everything comes at a cost: optimal lambda
will sacrifice the potential compression of the ViT.