PaPr: Training-Free One-Step Patch Pruning with Lightweight ConvNets for
  Faster Inference
Tanvir Mahmud Burhaneddin Yaman Chun-Hao Liu Diana Marculescu
As deep neural networks evolve from convolutional neural networks (ConvNets)
to advanced vision transformers (ViTs), there is an increased need to eliminate
redundant data for faster processing without compromising accuracy. Previous
methods are often architecture-specific or necessitate re-training, restricting
their applicability with frequent model updates. To solve this, we first
introduce a novel property of lightweight ConvNets: their ability to identify
key discriminative patch regions in images, irrespective of model's final
accuracy or size. We demonstrate that fully-connected layers are the primary
bottleneck for ConvNets performance, and their suppression with simple weight
recalibration markedly enhances discriminative patch localization performance.
Using this insight, we introduce PaPr, a method for substantially pruning
redundant patches with minimal accuracy loss using lightweight ConvNets across
a variety of deep learning architectures, including ViTs, ConvNets, and hybrid
transformers, without any re-training. Moreover, the simple early-stage
one-step patch pruning with PaPr enhances existing patch reduction methods.
Through extensive testing on diverse architectures, PaPr achieves significantly
higher accuracy over state-of-the-art patch reduction methods with similar FLOP
count reduction. More specifically, PaPr reduces about 70% of redundant patches
in videos with less than 0.8% drop in accuracy, and up to 3.7x FLOPs reduction,
which is a 15% more reduction with 2.5% higher accuracy.