Towards Image Semantics and Syntax Sequence Learning
Chun Tao Timur Ibrayev Kaushik Roy
Convolutional neural networks and vision transformers have achieved
outstanding performance in machine perception, particularly for image
classification. Although these image classifiers excel at predicting
image-level class labels, they may not discriminate missing or shifted parts
within an object. As a result, they may fail to detect corrupted images that
involve missing or disarrayed semantic information in the object composition.
On the contrary, human perception easily distinguishes such corruptions. To
mitigate this gap, we introduce the concept of "image grammar", consisting of
"image semantics" and "image syntax", to denote the semantics of parts or
patches of an image and the order in which these parts are arranged to create a
meaningful object. To learn the image grammar relative to a class of visual
objects/scenes, we propose a weakly supervised two-stage approach. In the first
stage, we use a deep clustering framework that relies on iterative clustering
and feature refinement to produce part-semantic segmentation. In the second
stage, we incorporate a recurrent bi-LSTM module to process a sequence of
semantic segmentation patches to capture the image syntax. Our framework is
trained to reason over patch semantics and detect faulty syntax. We benchmark
the performance of several grammar learning models in detecting patch
corruptions. Finally, we verify the capabilities of our framework in Celeb and
SUNRGBD datasets and demonstrate that it can achieve a grammar validation
accuracy of 70 to 90% in a wide variety of semantic and syntactical corruption
scenarios.