Measuring visual (dis)similarity between two or more instances within a data distribution is a fundamental task in many applications, specially in image retrieval. Theoretically, non-metric distances are able to generate a more complex and accurate similarity model than metric distances, provided that the non-linear data distribution is precisely captured by the similarity model. In this work, we analyze a simple approach for deep learning networks to be used as an approximation of non-metric similarity functions and we study how these models generalize across different image retrieval datasets. For humans, deciding whether two images are visually similar or not is, to some extent, a natural task. However, in computer vision, this is a challenging problem and algorithms do not always succeed in matching pictures that contain similar-looking elements. This is mainly because of the well-known semantic gap problem, which refers to the difference or gap between low-level image pixels and high-level semantic concepts. Estimating visual similarity is a fundamental task that seeks to break this semantic gap by accurately evaluating how alike two or more pictures are. Visual similarity is crucial for many computer vision areas including image retrieval, image classification and object recognition, among others.Given a query image, content-based image retrieval systems rank pictures in a dataset according to how similar they are with respect to the input. This can be broken into two fundamental tasks: 1) computing meaningful image representations that capture the most salient visual information from pixels and 2) measuring accurate visual similarity between these image representations to rank images according to a similarity score.In the last years, several methods to represent visual information from raw pixels in images have been proposed, first by designing handcrafted features such as SIFT BID22 , then by compacting these local features into a single global image descriptor using different techniques such as Fisher Vectors BID28 and more recently by extracting deep image representations from neural networks BID1 ). However, once two images are described by feature vectors, visual similarity is commonly measured by computing a standard metric between them. Although regular distance metrics, such as Euclidean distance or cosine similarity, are fast and easy to implement, they do not take into account the possible interdependency within the dataset, which means that even if a strong nonlinear data dependency is occurring in the visual collection, they might not be able to capture it. This suggests that learning a similarity estimation directly from visual data can improve the performance on image retrieval tasks, provided that the likely nonlinearity dependencies within the dataset are precisely learned by the similarity function.Visual similarity learning is closely related to distance metric learning. Traditionally, distance metric learning algorithms were based on linear metrics such as the Mahalanobis distance. However, if the visual data presents any nonlinear interdependency, better results are expected when using nonlinear approaches. According to some studies BID41 , standard metric axioms are not valid for human perception of visual similarity and hence, visual similarity functions should not necessarily satisfy distance metric conditions. Deep learning-based similarity learning methods are mostly focused on learning an optimal mapping from pixels to a linear space in which Euclidean distance can be applied. Instead, we propose a simple approach based on neural networks to learn a non-metric similarity score in the feature space. Figure 1: System overview. The feature extraction block computes visual representations of images whereas the visual similarity block estimates a similarity score using a neural network. Figure 1 shows an overview of the proposed approach. By training a deep learning model, we can estimate a visual similarity function that outperforms methods based on standard metric computations. One convolutional neural network extracts image representations from input images, while a second neural network computes the visual similarity score. The visual similarity neural network is trained using both pairs of similar and dissimilar images in three stages. The output score of the similarity network can be directly applied as a similarity estimation to rank images in an image retrieval task. Experimental results on standard datasets show that our network is able to discriminate when a pair of images is similar or dissimilar and improve standard metrics score on top of that. Four different configurations A-D for the similarity neural network are proposed. We compare the performance of each one during Stage 1, when the network is trained with the standard cosine similarity measurement. If s l is the network score and y l is the cosine similarity of the l-th pair with l = 1.. L, we evaluate each network by computing the mean squared error, MSE, and the correlation coefficient, ρ, as: DISPLAYFORM0 where µ s and σ s are the mean and standard deviation of the vector of network scores s, and µ y and σ y are the mean and standard deviation of the vectors of cosine similarities y.Results are shown in TAB0 . Unsurprisingly, the configuration with bigger number of parameters, C, achieves the best MSE and ρ results, both in training and validation sets. However, the performance of networks B and D is very close to the performance of network C. As network B requires only 21 million parameters and network C requires 76 million parameters, we keep configuration B as our default architecture for the rest of the experiments. We have presented a method for learning visual similarity directly from visual data. Instead of using a rigid metric distance, such as the standard cosine similarity, we propose to train a neural network model to learn a similarity estimation between a pair of visual representations previously extracted from input images. Our method outperforms state-of-the-art approaches based on rigid distances in standard image retrieval collection of images and experimental results showed that learning a nonmetric visual similarity function is beneficial in image retrieval tasks provided that a small subset of images of the same domain are available during training. Standard image retrieval techniques that are commonly applied after cosine similarity computation, such as query expansion or image re-ranking, might also be applied on top of the similarity network. Finally, we end with an open question, which is the subject of planned future work, concerning efficient computation of exact or approximate K-nearest neighbours based on the learned network similarity function.
Similarity network to learn a non-metric visual similarity estimation between a pair of images The authors propose learning similarity measure for visual similarity and obtain by this an improvement in very well-known datasets of Oxford and Paris for image retrival. The paper argues that it is more suitable to use non-metric distances instead of metric distances.