We present a deep learning-based method for super-resolving coarse (low-resolution) labels assigned to groups of image pixels into pixel-level (high-resolution) labels, given the joint distribution between those low- and high-resolution labels. This method involves a novel loss function that minimizes the distance between a distribution determined by a set of model outputs and the corresponding distribution given by low-resolution labels over the same set of outputs. This setup does not require that the high-resolution classes match the low-resolution classes and can be used in high-resolution semantic segmentation tasks where high-resolution labeled data is not available. Furthermore, our proposed method is able to utilize both data with low-resolution labels and any available high-resolution labels, which we show improves performance compared to a network trained only with the same amount of high-resolution data.
 We test our proposed algorithm in a challenging land cover mapping task to super-resolve labels at a 30m resolution to a separate set of labels at a 1m resolution. We compare our algorithm with models that are trained on high-resolution data and show that 1) we can achieve similar performance using only low-resolution data; and 2) we can achieve better performance when we incorporate a small amount of high-resolution data in our training. We also test our approach on a medical imaging problem, resolving low-resolution probability maps into high-resolution segmentation of lymphocytes with accuracy equal to that of fully supervised models. Semantic image segmentation is the task of labeling each pixel in an input image X = {x ij } as belonging to one of L fine-scale application classes, Y = {y ij }, y ∈ {1, . . . , L}. In weakly supervised segmentation, instances in the training set only contain partial observations of the target ground truth labels, e.g., summary of class labels instead of pixel-level labels. We aim to solve a variant of this problem where coarse-scale, low-resolution accessory classes, Z = {z k }; z ∈ {1, . . . , N }, are defined for sets of pixels in the input images, where we are given the joint distribution P (Y, Z) between the accessory class labels and the application labels. Specifically, a training image X is divided into K sets B k , each with an accessory class label z k , and our models are trained to produce the high-resolution application labels y ij . For example, in Figure 1 , a high-resolution aerial image is shown alongside the low-resolution ground truth land cover map (defined over accessory classes) and the target high-resolution version (defined over application classes). We aim to derive the high-resolution land cover map based on the aerial image and low-resolution ground truth.Compared to other weakly supervised image segmentation techniques, the formulation of the problem we aim to solve is more general: it applies both to existing weakly supervised image segmentation problems, as well as to other problems with different characteristics of weak labels. The more general formulation is necessary for tasks such as land cover mapping from aerial imagery and lymphocyte segmentation from pathology imagery. In these applications, coarse labels do not necessarily match the fine-scale labels, as shown in Figure 1 . The distinction between the fine-scale application and coarse-scale accessory classes is necessary for situations in which the ground-truth information that is known about an image does not match with the application classes that we aim to Figure 1 : An Illustration of land cover data and label super-resolution. Our method takes an input image (x) with low-resolution labels (z) and outputs a set of super-resolved label predictions (y), utilizing the statistical descriptions between low-resolution and high-resolution labels (Appendix B) e.g., one low-resolution class designates areas of low-intensity development, with 20% to 49% of impervious surfaces (such as houses or roads).label the image with, but instead suggests a distribution over the application labels. State-of-the-art methods for weakly supervised semantic segmentation exploit the structure of weak labels in ways that are not applicable in our examples: we cannot create bounding boxes around land cover object instances BID7 ; Papandreou et al. FORMULA0 ) -we consider data that is generally given at scales much larger than the objects being segmented and does not carry foreground-background morphology -nor use coarse approximations of ground-truth segmentation (Krähenbühl & Koltun (2011); BID13 ). Other work attempts to match a class "density function" to weak labels (Lempitsky & Zisserman (2010) ), but it mainly targets localization and enumeration of small foreground objects with known sizes. Existing Weak supervision approaches also often involve expensive steps in inference, such as CRFs or iterative evaluation BID3 ), which are impractical on large datasets. At the same time, thorough analyses of training algorithms only exist for models that are not sufficiently expressive for the applications we consider (Yu et al. (2013) ). While our formulation of the problem allows us to specifically address the previously mentioned land cover mapping and lymphocyte segmentation, it can also be applied to more traditional segmentation tasks such as foreground/background segmentation as we explore in Appendix. F.Our proposed method is illustrated in FIG0 . Briefly, a standard segmentation network will output probabilistic estimates of the application labels. Our methodology summarizes these estimates over the sets B k , which results in an estimated distribution of application labels for each set. These distributions can then be compared to the expected distribution from the accessory (low-resolution) labels using standard distribution distance metrics. This extension is fully differentiable and can thus be used to train image segmentation neural networks end-to-end from pairs of images and coarse labels.Land cover mapping from aerial imagery is an important application in need of such methodology. Land cover maps are essential in many sustainability-related applications such as conservation planning, monitoring habitat loss, and informing land management. In Section 3.1 we describe land cover mapping in detail and show how our method creates high-resolution land cover maps solely from high-resolution imagery low-resolution labels, at an accuracy similar to that of models trained on high-resolution labels. We further show how to train models with a combination of low-and highresolution labels that outperform the high-res models in transfer learning tasks. As low-resolution labels are much easier to collect, and indeed exist over a much wider geographic area in our land cover mapping application, the ability to combine low-and high-resolution labels is an important feature of our proposed methods.In a second example (Section 3.2), we segment tumor infiltrating lymphocytes from high-resolution (gigapixel) pathology images. Understanding the spatial distribution of immune cells, such as lymphocytes in pathology images, is fundamental for immunology and the treatment of cancer BID10 ; Thorsson et al. (2018) ). Here, coarse labels are probabilities of lymphocyte infiltration (having two or more lymphocytes) on 100×100 pixel regions, given by an automatic classifier (Saltz FORMULA0 ). Our super-resolution model trained on coarse labels performs the same as a lymphocyte classifier trained with high-resolution (cell-level) supervision (Hou et al. (2018) ).To summarize, as our first contribution, we propose a label super-resolution network which utilizes the distribution of high-resolution labels suggested by given low-resolution labels, based on visual cues in the input images, to derive high-resolution label predictions consistent to the input image. Our second contribution is that we evaluate our method extensively on the application of land cover segmentation and conclude that when there are not enough representative high-resolution training data, our method is much more robust than a model trained on high-resolution training data only, since our method utilizes more training data with weak labels. We show the generality of our method on the lymphocyte segmentation task and the task of segmenting foreground given object bounding boxes (in Appendix F). We proposed a label super-resolution network which is capable of deriving high-resolution labels, given low-resolution labels that do not necessarily match the targeting high-resolution labels in a one-to-one manner -we only assume that the joint distribution between the low-resolution and highresolution classes is known. In particular, we train a network to predict high-resolution labels, minimizing the distance/divergence between two distributions: distribution of predicted high-resolution labels and expected distribution suggested by the low-resolution labels. We applied our method in two real-world applications where high res labels are very expensive to obtain compared to low res labels, and achieved similar or better results compared to the conventional fully supervised methods trained on high-resolution labels. We also show how combining low and high res labels leads to better generalization to out-of-sample test sets.Although the main assumption of the model is that the joint distribution over coarse and fine labels is known, the model is in fact robust to errors in estimates of these distributions, as we discuss in Appendix F. There we show that these joint distributions can be acquired or inferred in a variety of ways, thus making label super-resolution widely applicable, including beyond computer vision.
Super-resolving coarse labels into pixel-level labels, applied to aerial imagery and medical scans. A method to super-resolve coarse low-res segmentation labels if the joint distribution of low-res and high-res labels are known.