We study the problem of learning to map, in an unsupervised way, between domains $A$ and $B$, such that the samples $\vb \in B$ contain all the information that exists in samples $\va\in A$ and some additional information. For example, ignoring occlusions, $B$ can be people with glasses, $A$ people without, and the glasses, would be the added information. When mapping a sample $\va$ from the first domain to the other domain, the missing information is replicated from an independent reference sample $\vb\in B$. Thus, in the above example, we can create, for every person without glasses a version with the glasses observed in any face image. 

 Our solution employs a single two-pathway encoder and a single decoder for both domains. The common part of the two domains and the separate part are encoded as two vectors, and the separate part is fixed at zero for domain $A$. The loss terms are minimal and involve reconstruction losses for the two domains and a domain confusion term. Our analysis shows that under mild assumptions, this architecture, which is much simpler than the literature guided-translation methods, is enough to ensure disentanglement between the two domains. We present convincing results in a few visual domains, such as no-glasses to glasses, adding facial hair based on a reference image, etc. In the problem of unsupervised domain translation, the algorithm receives two sets of samples, one from each domain, and learns a function that maps between a sample in one domain to the analogous sample in the other domain BID37 BID5 BID28 BID29 BID10 BID11 BID38 b; BID26 . The term unsupervised means, in this context, that the two sets are unpaired.In this paper, we consider the problem of domain B, which contains a type of content that is not present at A. As a running example, we consider the problem of mapping between a face without eyewear (domain A) to a face with glasses (domain B). While most methods would map to a person with any glasses, our solution is guided and we attach to an image a ∈ A, the glasses that are present in a reference image b ∈ B.In comparison to other guided image to image translation methods, our method is considerably simpler. It relies on having a latent space with two parts: (i) a shared part that is common to both A and B, and (ii) a specific part that encodes the added content in B. By setting the second part to be the zero vector for all samples in A, a disentanglement emerges. Our analysis shows that this Table 1 : A comparison to other unsupervised guided image to image translation methods.† k = 5 is the number of pre-segmented face parts.‡ Used for domain confusion, not on the output.MUNIT EG-UNIT BID30 DRIT BID27 PairedCycleGAN (Chang' domains. The networks are of four types: encoders, which map images to a latent space, generators (also known as decoders), which generate images from a latent representation, discriminators that are used as part of an adversarial loss, and other, less-standard, networks.It is apparent that our method is considerably simpler than the literature methods. The main reason is that our method is based on the emergence of disentanglement, as detailed in Sec. 4. This allows us to to train with many less parameters and without the need to apply excessive tuning, in order to balance or calibrate the various components of the compound loss.The MUNIT architecture by , like our architecture, employs a shared latent space, in addition to a domain specific latent space. Their architecture is not limited to two domains 1 and unlike ours, employs separate encoders and decoders for the various domains. The type of guiding that is obtained from the target domain in MUNIT is referred to as style, while in our case, the guidance provides content. Therefore , MUNIT, as can be seen in our experiments, cannot add specific glasses, when shifting from the no-glasses domain to the faces with eyewear domain.The EG-UNIT architecture by BID30 presents a few novelties, including an adaptive method of masking-out a varying set of the features in the shared latent space. In our latent representation of domain A, some of the features are constantly zero, which is much simpler. This method also focuses on guiding for style and not for content, as is apparent form their experiments.The very recent DRIT work by BID27 learns to map between two domains using a disentangled representation. Unlike our work, this work seems to focus on style rather than content. The proposed solution differs from us in many ways: (1) it relies on two-way mapping, while we only map from A to B. (2) it relies on shared weights in order to ensure that the common representation is shared. (3) it adds a VAE-like BID24 statistical characterization of the latent space, which results in the ability to sample random attributes. As can be seen in Tab. 1, the solution of BID27 is considerably more involved than our solution.DRIT (and also MUNIT) employ two different types of encoders that enforce a separation of the latent space representations to either style or content vectors. For example, the style encoder , unlike the content encoder, employs spatial pooling and it also results in a smaller representation than the content one. This is important, in the context of these methods, in order to ensure that the two representations encode different aspects of the image. If DRIT or MUNIT were to use the same type of encoder twice, then one encoder could capture all the information, and the image-based guiding (mixing representations from two images) would become mute. In contrast, our method (i) does not separate style and content, and (ii) has a representation that is geared toward capturing the additional content.The work most similar to us in its goal, but not in method, is the PairedCycleGAN work by BID7 . This work explores the single application of applying the makeup of a reference face to a source face image. Unfortunately, the method was only demonstrated on a proprietary unshared dataset and the code is also not publicly available, making a direct comparison impossible at this time. The method itself is completely different from ours and does not employ disentanglement. Instead, a generator with two image inputs is used to produce an output image, where the makeup is transfered between the input images, and a second generator is trained to remove makeup. The generation is done separately to k = 5 pre-segmented facial regions, and the generators do not employ an encoder-decoder architecture.Lastly, there are guided methods, which are trained in the supervised domain, i.e., when there are matches between domain A and B. Unlike the earlier one-to-one work, such as pix2pix BID22 , these methods produce multiple outputs based on a reference image in the target domain. Examples include the Bicycle GAN by , who also applied, as baseline in their experiments, the methods of BID2 BID15 .Other Disentanglement Work InfoGAN BID9 learns a representation in which, due to the statistical properties of the representations, specific classes are encoded as a one-hot encoding of part of the latent vector. In the work of ; BID16 , the representation is disentangled by reducing the class based information within it. The separate class based information is different in nature from our multi-dimensional added content. BID6 , which builds upon BID16 , performs guided image to image translation, but assumes the availability of class based information, which we do not. When converting between two domains, there is an inherent ambiguity that arises from the domainspecific information in the target domain. In guided translation, the reference image in the target domain provides the missing information. Previous work has focused on the missing information that is highly tied to the texture of the image. For example, when translating between paintings and photos, DRIT adds considerable content from the reference photo. However, this is unstructured content, which is not well localized and is highly related to subsets of the image patches that exist in the target domain. In addition, the content from the reference photo that is out of the domain of paintings is not guaranteed to be fully present in the output.Our work focuses on transformations in which the domain specific content is well structured, and guarantees to replicate all of the domain specific information from the reference image. This is done using a small number of networks and a surprisingly simple set of loss terms, which, due to the emergence of a disentangled representation, solves the problem convincingly. In this section we provide notations and terminology that are were not introduced in Sec. 4 but are necessary for the proofs of the claims in this section.We say that three random variables (discrete or continuous) X 1 , X 2 , X 3 form a Markov chain, indicated with DISPLAYFORM0 The Data Processing Inequality (DPI) for a Markov chain X 1 → X 2 → X 3 ensures that I(X 1 ; X 3 ) ≤ min (I(X 1 ; X 2 ), I(X 2 ; X 3 )). In particular, it holds for X 2 = f (X 1 ) and X 3 = g(X 2 ), where f, g are deterministic processes.We denote by x ∼ log N (µ, σ 2 ) a random variable that is distributed by a log-normal distribution, i.e., log x ∼ N (µ, σ 2 ). We consider that the mean and variance of a log-normal distribution log N (µ, σ 2 ) are exp(µ + σ 2 /2) and (exp(σ 2 ) − 1) exp(2µ + σ 2 ) respectively. We denote by W U := (W k,j · U k,j ) k≤m,j≤m the Hadamard product of two matrices W , U ∈ R m×n . For a given vector x ∈ R m , we denote dim(x) := m and for a matrix W ∈ R m×n , we denote dim(W ) := mn. In addition, we denote x 2 = x x = (x 2 1 , . . . , x 2 m ) and DISPLAYFORM1
An image to image translation method which adds to one image the content of another thereby creating a new image. This paper tackles the task of content transfer, with the novalty being on the loss.