EVP: Enhanced Visual Perception using Inverse Multi-Attentive Feature
  Refinement and Regularized Image-Text Alignment
Mykola Lavreniuk Shariq Farooq Bhat Matthias Müller Peter Wonka
This work presents the network architecture EVP (Enhanced Visual Perception).
EVP builds on the previous work VPD which paved the way to use the Stable
Diffusion network for computer vision tasks. We propose two major enhancements.
First, we develop the Inverse Multi-Attentive Feature Refinement (IMAFR) module
which enhances feature learning capabilities by aggregating spatial information
from higher pyramid levels. Second, we propose a novel image-text alignment
module for improved feature extraction of the Stable Diffusion backbone. The
resulting architecture is suitable for a wide variety of tasks and we
demonstrate its performance in the context of single-image depth estimation
with a specialized decoder using classification-based bins and referring
segmentation with an off-the-shelf decoder. Comprehensive experiments conducted
on established datasets show that EVP achieves state-of-the-art results in
single-image depth estimation for indoor (NYU Depth v2, 11.8% RMSE improvement
over VPD) and outdoor (KITTI) environments, as well as referring segmentation
(RefCOCO, 2.53 IoU improvement over ReLA). The code and pre-trained models are
publicly available at https://github.com/Lavreniuk/EVP.