Exploiting Object-based and Segmentation-based Semantic Features for
  Deep Learning-based Indoor Scene Classification
Ricardo Pereira Luís Garrote Tiago Barros Ana Lopes Urbano J. Nunes
Indoor scenes are usually characterized by scattered objects and their
relationships, which turns the indoor scene classification task into a
challenging computer vision task. Despite the significant performance boost in
classification tasks achieved in recent years, provided by the use of
deep-learning-based methods, limitations such as inter-category ambiguity and
intra-category variation have been holding back their performance. To overcome
such issues, gathering semantic information has been shown to be a promising
source of information towards a more complete and discriminative feature
representation of indoor scenes. Therefore, the work described in this paper
uses both semantic information, obtained from object detection, and semantic
segmentation techniques. While object detection techniques provide the 2D
location of objects allowing to obtain spatial distributions between objects,
semantic segmentation techniques provide pixel-level information that allows to
obtain, at a pixel-level, a spatial distribution and shape-related features of
the segmentation categories. Hence, a novel approach that uses a semantic
segmentation mask to provide Hu-moments-based segmentation categories' shape
characterization, designated by Segmentation-based Hu-Moments Features (SHMFs),
is proposed. Moreover, a three-main-branch network, designated by
GOS$^2$F$^2$App, that exploits deep-learning-based global features,
object-based features, and semantic segmentation-based features is also
proposed. GOS$^2$F$^2$App was evaluated in two indoor scene benchmark datasets:
SUN RGB-D and NYU Depth V2, where, to the best of our knowledge,
state-of-the-art results were achieved on both datasets, which present
evidences of the effectiveness of the proposed approach.