Instance embeddings are an efficient and versatile image representation that facilitates applications like recognition, verification, retrieval, and clustering. Many metric learning methods represent the input as a single point in the embedding space. Often the distance between points is used as a proxy for match confidence. However, this can fail to represent uncertainty which can arise when the input is ambiguous, e.g., due to occlusion or blurriness. This work addresses this issue and explicitly models the uncertainty by “hedging” the location of each input in the embedding space. We introduce the hedged instance embedding (HIB) in which embeddings are modeled as random variables and the model is trained under the variational information bottleneck principle (Alemi et al., 2016; Achille & Soatto, 2018). Empirical results on our new N-digit MNIST dataset show that our method leads to the desired behavior of “hedging its bets” across the embedding space upon encountering ambiguous inputs. This results in improved performance for image matching and classification tasks, more structure in the learned embedding space, and an ability to compute a per-exemplar uncertainty measure which is correlated with downstream performance. An instance embedding is a mapping f from an input x, such as an image, to a vector representation, z ∈ R D , such that "similar" inputs are mapped to nearby points in space. Embeddings are a versatile representation that support various downstream tasks, including image retrieval (Babenko et al., 2014) and face recognition (Schroff et al., 2015) .Instance embeddings are often treated deterministically, i.e., z = f (x) is a point in R D . We refer to this approach as a point embedding. One drawback of this representation is the difficulty of modeling aleatoric uncertainty (Kendall & Gal, 2017) , i.e. uncertainty induced by the input. In the case of images this can be caused by occlusion, blurriness, low-contrast and other factors.To illustrate this, consider the example in FIG7 . On the left , we show an image composed of two adjacent MNIST digits, the first of which is highly occluded. The right digit is clearly a 7, but the left digit could be a 1, or a 4. One way to express this uncertainty about which choice to make is to map the input to a region of space, representing the inherent uncertainty of "where it belongs".We propose a new method, called hedged instance embedding (HIB), which achieves this goal. Each embedding is represented as a random variable, Z ∼ p(z|x) ∈ R D . The embedding effectively spreads probability mass across locations in space, depending on the level of uncertainty. For example in Figure 1b , the corrupted image is mapped to a two-component mixture of Gaussians covering both the "17" and "47" clusters. We propose a training scheme for the HIB with a learnable-margin contrastive loss and the variational information bottleneck (VIB) principle (Alemi et al., 2016; BID1 . Figure 1: Unlike point embeddings, stochastic embeddings may hedge their bets across the space. When both "17" and "47 " are plausible, our 2-component Gaussian mixture embedding has the power to spread probability mass on clusters with clean "17" and "47" images. By contrast, the point embedding will choose to be close to one or the other of these points (or somewhere between).To evaluate our method, we propose a novel dataset, N-digit MNIST, which we will open source.Using this dataset, we show that HIB exhibits several desirable properties compared to point embeddings: (1) downstream task performance (e.g. recognition and verification) improves for uncertain inputs; (2) the embedding space exhibits enhanced structural regularity; and (3) a per-exemplar uncertainty measure that predicts when the output of the system is reliable. Hedged instance embedding is a stochastic embedding that captures the uncertainty of the mapping of an image to a latent embedding space, by spreading density across plausible locations. This results in improved performance on various tasks, such as verification and identification, especially for ambiguous corrupted input. It also allows for a simple way to estimate the uncertainty of the embedding that is correlated with performance on downstream tasks.There are many possible directions for future work, including experimenting with higherdimensional embeddings, and harder datasets. As an early look at these tasks, in the Appendix, Section C.3, we apply HIB towards cat and dog instance embedding directed towards identifying specific animals with 20D embeddings. It would also be interesting to consider the "open world" (or "unknown unknowns") scenario, in which the test set may contain examples of novel classes, such as digit combinations that were not in the training set (see e.g., Lakkaraju et al. FORMULA1 ; Günther et al. FORMULA1 ). This is likely to result in uncertainty about where to embed the input which is different from the uncertainty induced by occlusion, since uncertainty due to open world is epistemic (due to lack of knowledge of a class), whereas uncertainty due to occlusion is aleatoric (intrinsic, due to lack of information in the input), as explained in Kendall & Gal (2017) . Preliminary experiments suggest that η(x) correlates well with detecting occluded inputs, but does not work as well for novel classes. We leave more detailed modeling of epistemic uncertainty as future work.
The paper proposes using probability distributions instead of points for instance embeddings tasks such as recognition and verification. Paper proposes an alternative to current point embedding and a technique to train them. The paper proposes a model using uncertain-embeddings to extend deep learning to Bayesian applications