In this work, we face the problem of unsupervised domain adaptation with a novel deep learning approach which leverages our finding that entropy minimization is induced by the optimal alignment of second order statistics between source and target domains. We formally demonstrate this hypothesis and, aiming at achieving an optimal alignment in practical cases, we adopt a more principled strategy which, differently from the current Euclidean approaches, deploys alignment along geodesics. Our pipeline can be implemented by adding to the standard classification loss (on the labeled source domain), a source-to-target regularizer that is weighted in an unsupervised and data-driven fashion. We provide extensive experiments to assess the superiority of our framework on standard domain and modality adaptation benchmarks. Learning visual representations that are invariant across different domains is an important task in computer vision. Actually, data labeling is onerous and even impossible in some cases. It is thus desirable to train a model with full supervision on a source, labeled domain and then learn how to transfer it on a target domain, as opposed to retrain it completely from scratch. Moreover, the latter stage is actually not possible if the target domain is totally unlabelled: this is the setting we consider in our work. In the literature, this problem is known as unsupervised domain adaptation which can be regarded as a special semi-supervised learning problem, where labeled and unlabeled data come from different domains. Since no labels are available in the target domain, source-to-target adaptation must be carried out in a fully unsupervised manner. Clearly, this is an arguably difficult task since transferring a model across domains is complicated by the so-called domain shift [Torralba & Efros (2011) ]. In fact, while switching from the source to the target, even if dealing with the same K visual categories in both domains, different biases may arise related to several factors. For instance, dissimilar points of view, illumination changes, background clutter, etc.In the previous years, a broad class of approaches has leveraged on entropy optimization as a proxy for (unsupervised) domain adaptation, borrowing this idea from semi-supervised learning [Grandvalet & Bengio (2004) ]. By either performing entropy regularization [Tzeng et al. (2015) ; Carlucci et al. (2017) ; Saito et al. (2017) ], explicit entropy minimization [Haeusser et al. (2017) ], or implicit entropy maximization through adversarial training [Ganin & Lempitsky (2015) ; Tzeng et al. (2017) ], this statistical tool has demonstrated to be powerful for adaptation purposes.Alternatively, there exist methods which try to align the source to the target domain by learning an explicit transformation between the two so that the target data distribution can be matched to the one of the source one [Glorot et al. (2011); Kan et al. (2015) ; Shekhar et al. (2013) ; Gopalan & Li (2011); Gong et al. (2012a) ]. Within this paradigm, correlation alignment minimizes the distance between second order statistics computed in the form of covariance representations between features from the source a [Fernando et al. (2013) ; Sun et al. (2016) ; Sun & Saenko (2016) ].Apparently , correlation alignment and entropy minimization may seem two unrelated and approaches in optimizing models for domain adaptation. However, in this paper, we will show that this is not the case and, indeed, we claim that the two classes of approaches are deeply intertwined. In addition to formally discuss the latter aspect, we also obtain a solution for the prickly problem of hyperparameter validation in unsupervised domain adaptation. Indeed, one can construct a validation set out of source data but the latter is not helpful since not representative of target data. At the same time, due to the lack of annotations on the target domain, usual (supervised) validation techniques can not be applied.In summary, this paper brings the following contributions.1. We explore the two paradigms of correlation alignment and entropy minimization, by formally demonstrating that, at its optimum, correlation alignment attains the minimum of the sum of cross-entropy on the source domain and of the entropy on the target.2. Motivated by the urgency of penalizing correlation misalignments in practical terms, we observe that an Euclidean penalty, as adopted in [Sun et al. (2016); Sun & Saenko (2016) ], is not taking into account the structure of the manifold where covariance matrices lie in. Hence, we propose a different loss function that is inspired by a geodesic distance that takes into account the manifold's curvature while computing distances.3. When aligning second order statistics, a hyper-parameter controls the balance between the reduction of the domain shift and the supervised classification on the source domain. In this respect , a manual cross-validation of the parameter is not straightforward: doing it on the source domain may not be representative, and it is not possible to do on the target due to the lack of annotations. Owing to our principled connection between correlation alignment and entropy regularization, we devise an entropy-based criterion to accomplish such validation in a data-driven fashion.4. We combine the geodesic correlation alignment with the entropy-based criterion in a unique pipeline that we call minimal-entropy correlation alignment. Through an extensive experimental analysis on publicly available benchmarks for transfer object categorization, we certify the effectiveness of the proposed approach in terms of systematic improvements over former alignment methods and state-of-the-art techniques for unsupervised domain adaptation in general.The rest of the paper is outlined as follows. In Section 2, we report the most relevant related work as background material. Section 3 presents our theoretical analysis which inspires our proposed method for domain adaptation (Section 4). We report a broad experimental validation in Section 5. Finally, Section 6 draws conclusions. In this paper we carried out a principled connection between correlation alignment and entropy minimization, formally demonstrating that the optimal solution to the former problem gives for free the optimal solution of the latter. This improved knowledge brought us to two algorithmic advances. First, we achieved a more effective alignment of covariance operators which guarantees a superior performance. Second, we derived a novel cross-validation approach for the hyper-parameter λ so that we can obtain the maximum performance on the target, even not having access to its labels. These two components, when combined in our proposed MECA pipeline, provide a solid performance against state-of-the-art methods for unsupervised domain adaptation.L.J.P van der Maaten and G.E. Hinton. Visualizing high-dimensional data using t-sne. For the problem of (unsupervised) domain adaptation, a first class of methods aims at learning transformations which align feature representations in the source and target sets. For instance, in [Glorot et al. (2011) ] auto-encoders are exploited to learn common features. In [Kan et al. (2015) ], a bi-shifting auto-encoder (BSA) is instead intended to shift source domain samples into target ones and, similarly, other methods approach the same problem by means of techniques based on dictionary learning (as in [Shekhar et al. (2013)] ). Geodesic methods (such as [Gopalan & Li (2011); Gong et al. (2012a) ] aim at projecting source and target datasets on a common manifold in such a way that the projection already solves the alignment problem. The approaches [Gong et al. (2012b) ; Gopalan et al. FORMULA0 ] learns a smooth transition between the source and data manifold by means of Principal Components Analysis and Partial Least Squares, respectively. Inspired by the idea of adapting second order statistics between the two domains, [Sun et al. (2016); Fernando et al. (2013) ] propose a transformation to minimize the distance between the covariances of source and target datasets in order to, ultimately, achieve correlation alignment. Due to the well known properties of covariance operators, in some cases [Sun et al. (2016) ], the alignment can be written down in closed-form. But, since the latter operation can be prohibitively expensive in terms of computational cost, Sun & Saenko (2016) implements correlation alignment in an end-to-end fashion by means of backpropagation.A complementary family of approaches exploit the powerful statistical tool of entropy optimization in order to carry out adaptation. Indeed, the notion of association [Haeusser et al. (2017) ] is actually implementing explicit entropy minimization [Grandvalet & Bengio (2004) ] to align the target to the source embedding by navigating the data manifold by means of closed cyclic paths that interconnect instances belonging to the same objects' classes. In parallel, there are cases [Ganin & Lempitsky (2015) ; Tzeng et al. (2017) ] where minimax optimization is responsible for doing the following adversarial training. One seeks for feature representations that are effective for the primary visual recognition task being at the same time invariant while changing from source to target. The latter stage is implemented as the attempt of devising a random chance classifier which is asked to detect whether a given feature vector has been computed from a source or target data instance. Therefore, those approaches are implicitly promoting entropy maximization 3 at the classifier level. Finally, entropy regularization is accomplished in [Tzeng et al. (2015); Carlucci et al. (2017); Saito et al. (2017) ] as a complementary step to boost adaptation. Indeed, already established techniques for adaptation such as Batch Normalization [Ioffe & Szegedy (2015) ; Li et al. (2016) ] are applied in low-level layers to align the representations. On top of that, adaptation is refined at the end of the feature hierarchy by introducing a entropy-based regularizer on the target domain based. Practically, the latter exploits network's prediction to generate pseudo-labels [Lee FORMULA0
A new unsupervised deep domain adaptation technique which efficiently unifies correlation alignment and entropy minimization Improves the correlation alignment approach to domain adaptation by replacing the Euclidean distance with the geodesic Log-Euclidean distance between two covariance matices, and automatically selecting the balancing cost by the entropy on the target domain. Proposal for minimal-entropy correlation alignment, an unsupervised domain adaptation algorithm which links together entropy minimization and correlation alignment methods.