AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic
  Segmentation
Jiannan Ge Lingxi Xie Hongtao Xie Pandeng Li Xiaopeng Zhang Yongdong Zhang Qi Tian
A serious issue that harms the performance of zero-shot visual recognition is
named objective misalignment, i.e., the learning objective prioritizes
improving the recognition accuracy of seen classes rather than unseen classes,
while the latter is the true target to pursue. This issue becomes more
significant in zero-shot image segmentation because the stronger (i.e.,
pixel-level) supervision brings a larger gap between seen and unseen classes.
To mitigate it, we propose a novel architecture named AlignZeg, which embodies
a comprehensive improvement of the segmentation pipeline, including proposal
extraction, classification, and correction, to better fit the goal of zero-shot
segmentation. (1) Mutually-Refined Proposal Extraction. AlignZeg harnesses a
mutual interaction between mask queries and visual features, facilitating
detailed class-agnostic mask proposal extraction. (2) Generalization-Enhanced
Proposal Classification. AlignZeg introduces synthetic data and incorporates
multiple background prototypes to allocate a more generalizable feature space.
(3) Predictive Bias Correction. During the inference stage, AlignZeg uses a
class indicator to find potential unseen class proposals followed by a
prediction postprocess to correct the prediction bias. Experiments demonstrate
that AlignZeg markedly enhances zero-shot semantic segmentation, as shown by an
average 3.8% increase in hIoU, primarily attributed to a 7.1% improvement in
identifying unseen classes, and we further validate that the improvement comes
from alleviating the objective misalignment issue.