Manipulation and re-use of images in scientific publications is a recurring problem, at present lacking a scalable solution.   Existing tools for detecting image duplication are mostly manual or semi-automated, despite the fact that generating data for a learning-based approach is straightforward, as we here illustrate. This paper addresses the problem of determining if, given two images, one is a manipulated version of the other by means of certain geometric and statistical manipulations, e.g. copy, rotation, translation, scale, perspective transform, histogram adjustment, partial erasing, and compression artifacts. We propose a solution based on a 3-branch Siamese Convolutional Neural Network. The ConvNet model is trained to map images into a 128-dimensional space, where the Euclidean distance between duplicate (respectively, unique) images is no greater (respectively, greater) than 1. Our results suggest that such an approach can serve as tool to improve surveillance of the published and in-peer-review literature for image manipulation. We also show that as a byproduct the network learns useful representations for semantic segmentation, with performance comparable to that of domain-specific models. Duplicative data reporting in the biomedical literature is more prevalent than most people realize BID1 ). One common form of data duplication, regardless of intent, is the re-use of scientific images, across multiple publications or even within the same publication. In some cases, images are altered before being re-used BID1 ). Changing orientation, perspective or image statistics, introducing skew or crop, and deleting or inserting data into the original image plane are all ways in which image data may be altered prior to inappropriate introduction, or re-introduction, into the reporting of experimental outcomes BID13 ; BID4 ; BID2 ). While the scientific community has affirmatively recognized the need for preventing the incorporation of duplicative or flawed image data into the scientific record, a consistent approach to screening and identifying problematic image data has yet to be established BID11 BID12 ).Cases of image data duplication and/or manipulation have often been detected by fellow scientists 1 or by editorial staff during the manuscript review process. Efforts to move towards automation include tools developed to isolate regions of manipulation within images already flagged as suspicious BID9 ). However , current methods for identifying duplicative and/or manipulated images largely rely on individual visual identification with accompanying application of qualitative similarity measures 2 . Given the rate at which the scientific literature is expanding, it is not feasible for all cases of potential image manipulation to be detected by human eyes. Thus, there is a continued need for automated tools to detect potential duplications, even in the presence of manipulation, to allow for more focused, thorough evaluation of this smaller errant image candidate pool. Such a tool would be invaluable to scientists and research staff on many levels, from figure screening as a step in improving raw data maintenance and manuscript preparation at the laboratory level BID13 ), to the routine screening by journal editorial staff of submitted manuscripts prior to the peer-review process BID11 BID5 ).The general problem of detecting similar images has been well studied in the field of computer vision (e.g. BID17 ; BID16 ; BID15 ). The one application that stands out is determining if two given faces are of the same person, where recent breakthroughs in deep Convolutional Neural Networks have allowed rapid progress BID14 ).In this paper, we apply modern methods in metric learning to address the problem of detecting image manipulation and re-use in scientific work. Specifically, we train a ConvNet to learn an image embedding such that images with the same original content, albeit altered through a common set of image manipulations, appear close to each other in the embedding space. We train this model on a large corpus of simulated image manipulations, and test on a small set of manipulated images from known instances of image duplication/manipulation 3 . To our knowledge, this is the first application of deep learning to the detection of image re-use in the scientific literature, although there have been works on the area of detecting image manipulation (e.g. BID0 ).We focus on the domain of biological images, since we have easy access to one such dataset, but naturally the model to be described is agnostic to the image domain. We test the learned forensic representation not only on new/unseen synthetic and real data for the problem of duplicate-detection, but also on a somewhat unrelated area: semantic segmentation. We show that the features learned in the convolution layers of the siamese network can be readily plugged into a pixel classifier, yielding results comparable with those of state-of-the art, domain-specific architectures. We have demonstrated that siamese networks have the potential to improve surveillance of the published and in-peer-review literature for duplicated images. This approach may not prove accurate enough to definitively determine image duplication, but rather could serve to narrow down the pool of images which are subjected to further review.We found that most errors in the real-world test set involved histogram/contrast alterations that are difficult to simulate, or scale changes beyond those the network was trained to detect. We will continue to explore synthetic manipulations as a way to improve accuracy of the algorithm.As indicated by the self-similarity matrices in Figure 3 , improvements are needed when different images are from the same category. We will improve the training procedure to sample more of these hard cases.One of the main roadblocks to this research is the lack of a public, large-scale database of image manipulation cases on which to further test the model. The challenge here is not only of generating one such dataset, but also of securing the proper permissions to release the data, given the legal issues involved. We are continually expanding our dataset and will make it available as soon as possible.The application to semantic segmentation was discovered somewhat by chance in an attempt to circumvent issues with the U-Net, mainly the need for a large corpus of annotations and difficulty setting hyperparameters. In contrast, the representation provided by the forensic siamese net is quite easy to deploy in conjunction with a random forest classifier.
A forensic metric to determine if a given image is a copy (with possible manipulation) of another image from a given dataset. Introduces the siamese network to identify duplicate and copied/modified images, which can be used to improve surveillance of the published and in-peer-review literature. The paper presents an application of deep convolutional networks for the task of duplicate image detection This work addresses the problem of finding duplicate/near duplicate images from biomedical publications and proposes a standard CNN and loss functions and apply it to this field.