Addressing Discrepancies in Semantic and Visual Alignment in Neural
  Networks
Natalie Abreu Nathan Vaska Victoria Helus
For the task of image classification, neural networks primarily rely on
visual patterns. In robust networks, we would expect for visually similar
classes to be represented similarly. We consider the problem of when
semantically similar classes are visually dissimilar, and when visual
similarity is present among non-similar classes. We propose a data augmentation
technique with the goal of better aligning semantically similar classes with
arbitrary (non-visual) semantic relationships. We leverage recent work in
diffusion-based semantic mixing to generate semantic hybrids of two classes,
and these hybrids are added to the training set as augmented data. We evaluate
whether the method increases semantic alignment by evaluating model performance
on adversarially perturbed data, with the idea that it should be easier for an
adversary to switch one class to a similarly represented class. Results
demonstrate that there is an increase in alignment of semantically similar
classes when using our proposed data augmentation method.