Theories in cognitive psychology postulate that humans use similarity as a basis
 for object categorization. However, work in image classification generally as-
 sumes disjoint and equally dissimilar classes to achieve super-human levels of
 performance on certain datasets. In our work, we adapt notions of similarity using
 weak labels over multiple hierarchical levels to boost classification performance.
 Instead of pitting clustering directly against classification, we use a warm-start
 based evaluation to explicitly provide value to a clustering representation by its
 ability to aid classification. We evaluate on CIFAR10 and a fine-grained classifi-
 cation dataset to show improvements in performance with the procedural addition
 of intermediate losses and weak labels based on multiple hierarchy levels. Further-
 more, we show that pretraining AlexNet on hierarchical weak labels in conjunc-
 tion with intermediate losses outperforms a classification baseline by over 17% on
 a subset of Birdsnap dataset. Finally, we show improvement over AlexNet trained
 using ImageNet pre-trained weights as initializations which further supports our 
 claim of the importance of similarity. Similarity is one of the bases of object categorization in humans BID14 . Theories of perceptual categorization in cognitive psychology postulate that humans construct categories by grouping similar stimuli to construct one or several prototypes. New instances are then labeled as the category that they are most similar to. For instance, when one categorizes a specific animal as a dog, they are saying that it is more similar to previously observed dogs than it is to all other objects. This view of categorization better explains the often fuzzy boundaries between real-life classes, where a new object may be equally similar to multiple prototypes. For example, while a dolphin is technically a mammal, its visual appearance is similar to fish, resulting in a lot of people misclassifying it.While research in cognitive psychology on object categorization might indicate that similarity should play a central role in object classification in humans, image classification in computer vision seems to do pretty well without using it. The task is commonly interpreted as one of classifying an image into one of multiple classes that are assumed to be disjoint and equally dissimilar. The use of softmax-based loss functions assumes that the classes are disjoint, while the use of one-hot labels assumes that classes are equally dissimilar. Despite those strong assumptions, which seem to violate human notions of categories, image classification has shown remarkable progress, achieving superhuman performance on multiple datasets BID6 BID3 . Far from being an anomaly, state-of-the-art models across image classification benchmark datasets use losses, such as cross-entropy loss, that make the explicit assumption of disjoint classes. However, BID4 notes that the predictions of ensembles of image classification models produces soft labels that "define a rich similarity structure over the data" despite the predictions of individual models not capturing this structure.Can similarity-based metrics improve classification performance in convolutional neural networks? Previous work has tried to answer this question in different problems such as metric learning BID0 , clustering BID16 , and hierarchical classification BID11 . We focus on applications of similarity in the context of convolutional neural networks such as BID5 who use a contrastive loss to perform end-to-end clustering using weak labels. While they show impressive performance on MNIST BID10 and CIFAR-10 (Krizhevsky & Hinton, 2009) , their method does not scale well to more complex datasets. BID13 propose a new loss function, magnet loss, to learn a representational space where distance corresponds to similarity. They show that clustering in this space allows them to achieve state-of-the-art performance on multiple fine-grained classification datasets.1 However, they initialize their model with pre-trained weights on ImageNet BID15 , so it is not clear whether their model is learning a good representation, or if it is finding a good mapping between already learned representations and the labels. BID20 pose the problem as one of hierarchical classification and propose a loss based on an ultra-metric tree representation of a hierarchy over the labels. While their loss has interesting properties, it only outperforms classification losses for datasets with a small number of instances per class. In this work, we use a contrastive loss to improve the classification performance of randomly initialized convolutional neural networks on a fine-grained classification task.What measures of similarity should we teach our model? We represent the relations between our classes in a hierarchy where the labels are all leaf nodes. Therefore, there are two kinds of similarities that we want our model to capture. The first is intra-class similarity-all cats are similar to each other and they are dissimilar to other animals. The second is inter-class similarity-dogs and cats are more similar to each other than they are to non-living objects. For a more complex hierarchy, our model should learn similarity at different levels of the class hierarchy. BID20 use their hierarchical loss function to learn those similarities, however, they observe that reducing the similarity between two classes across all levels of the hierarchy to a single value biases the model towards correct classification at the higher levels of the hierarchy resulting in poor classification performance. BID13 also observe that applying classification losses to the final layer reduces the entire representation of each class to a single scalar value which destroys intra-and inter-class variation. However, these are the same variations that we want our model to capture. We overcome those limitations by explicitly training the model to capture different grains of similarity at different levels of the network through applying an intermediate pairwise contrastive loss at those levels. In this manner, we can use hierarchical information, such as species of birds having the same coarse-grained category of bird while having different fine-grained labels. Despite being able to encode different levels of similarity, a contrastive loss does not require an explicit hierarchy; we only need weak labels for pairs of instances.How can we evaluate the representations learned by a clustering algorithm? Previous work has always tried to compete against classification using metrics biased towards the latter task. While some researchers have used hierarchical metrics BID20 or qualitative measures BID13 to evaluate the quality of their representations, their primary evaluation criteria has consistently been the classification accuracy of their model. We propose another way to evaluate the clustering representations by using the clustering model as a "warm start" from which they can train on classification. The intuition is that if the clustering model learned a representations that captures similarity in a space, it would be be at an advantage when it is fine-tuned for classification.We propose the use of a contrastive loss at different layers of a convolutional neural network to learn a notion of similarity across a set of labels. We also propose the use of the accuracy of a model pre-trained for clustering and fine-tuned for classification as an evaluation metric for clustering algorithms. In this work, we argue that similarity-based losses can be applied as a warm start to boost the performance of image classification models. We show that applying a pairwise contrastive loss to intermediate layers of the network results in better clustering performance, as well as better finetuned classification performance. Furthermore, we demonstrate how the pairwise loss can be used to train a model using weak hierarchical labels.We find that training with hierarchical labels results in the highest performance beating models with more parameters and approaches the performance of models pre-trained with ImageNet weights. This is a very significant finding since ImageNet contains multiple bird classes, so a model pretrained with ImageNet has seen many more birds than a model pre-trained with pairwise constraints on Birdsnap34. Nevertheless, it only outperforms a cluster-based warm-start model by just 10%. Regardless, we also show that applying our approach to ImageNet weights still results in a boost of 1.27%. This supports our claim that similarity-based metrics can improve classification performance, but it suggests that the gains we expect decrease if we start of with a good representational space.We hope to expand this in future work in multiple ways. We plan on extending our approach to the entire Birdsnap dataset, as well as to other fine-grained classification datasets. We also hope to perform more extensive analysis of the quality of the embedding spaces learned as it is obvious that the use of the Hungarian algorithm does not accurately reflect the performance of the clustering model.
Cluster before you classify; using weak labels to improve classification  Proposes using a clustering based loss function at multiple levels of a deepnet as well as using hierarchical structure of the label space to train better representations. This paper uses hierarchical label information to impose additional losses on intermediate representations in neural network training.