Window Attention is Bugged: How not to Interpolate Position Embeddings
Daniel Bolya Chaitanya Ryali Judy Hoffman Christoph Feichtenhofer
Window attention, position embeddings, and high resolution finetuning are
core concepts in the modern transformer era of computer vision. However, we
find that naively combining these near ubiquitous components can have a
detrimental effect on performance. The issue is simple: interpolating position
embeddings while using window attention is wrong. We study two state-of-the-art
methods that have these three components, namely Hiera and ViTDet, and find
that both do indeed suffer from this bug. To fix it, we introduce a simple
absolute window position embedding strategy, which solves the bug outright in
Hiera and allows us to increase both speed and performance of the model in
ViTDet. We finally combine the two to obtain HieraDet, which achieves 61.7 box
mAP on COCO, making it state-of-the-art for models that only use ImageNet-1k
pretraining. This all stems from what is essentially a 3 line bug fix, which we
name "absolute win".