Within-class variation in a high-dimensional dataset can be modeled as being on a low-dimensional manifold due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.). We desire a method for learning a representation of the manifolds induced by identity-preserving transformations that can be used to increase robustness, reduce the training burden, and encourage interpretability in machine learning tasks. In particular, what is needed is a representation of the transformation manifold that can robustly capture the shape of the manifold from the input data, generate new points on the manifold, and extend transformations outside of the training domain without significantly increasing the error. Previous work has proposed algorithms to efficiently learn analytic operators (called transport operators) that define the process of transporting one data point on a manifold to another.   The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data. The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space. These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification. The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation.   These results are shown on stylized constructions using the classic swiss roll data structure and in demonstrations of transfer learning in a data augmentation task for few-shot image classification. We also propose the use of transport operators for injecting transformations into new data examples which allows for realistic image animation. While significant progress has been made in training classifiers that can effectively discriminate between thousands of classes, the increasing classifier complexity obfuscates the reasoning behind class selection and requires large training datasets to capture variations within each class. In many settings, the within-class variation in a high-dimensional dataset can be modeled as being lowdimensional due to the constraints of the physical processes producing that variation (e.g., translation, illumination, etc.). When these variations are within a linear subspace, classic techniques such as principal component analysis (PCA) can be used to efficiently capture the transformations within the data. However, the manifold hypothesis states that in many cases of interest (e.g., images, sounds, text), the within-class variations lie on or near a low-dimensional nonlinear manifold BID5 . In the neuroscience literature there is a hypothesis that the manifold nature of these variations is explicitly exploited by hierarchical processing stages to untangle the representations of different objects undergoing the same physical transformations (e.g., pose) BID7 .By learning manifold representations of identity-preserving transformations on a subset of classes or data points, we gain the ability to transfer these natural variations to previously unseen data. A generative model of the transformation manifold trained on a dataset rich in variation can be used to transfer knowledge of those variations to previously unseen domains. This can increase robustness and reduce the training burden for machine learning tasks as well as enable the generation of novel examples. Additionally , an explicit understanding of the transformations occuring within a dataset provides interpretibility of machine learning tasks that is typically unavailable. One can view this approach as a variant of pattern theory that seeks transformations that operate on representational primitives BID15 , but with an explicit unsupervised learning of the low-dimensional manifold structure of many real-world datasets.There are a large number of "manifold learning" algorithms that have been introduced in the literature to discover manifold structure from data. The most common approach to this task is to perform an embedding of the original data points after performing a transformation to preserve either local or global properties of the manifold (e.g., local neighborhood relationships, global geodesic distances, etc.) BID32 BID29 BID37 BID1 . Unfortunately, such approaches capture manifold structure through a transformation of the data points themselves into a lower dimensional space and therefore are not suitable for the desired tasks. Specifically, there is no generative model of the data in the original high-dimensional space, meaning that the inferred manifold structure is not transferable to other data classes, is not amenable to strong interpolation/extrapolation, and does not provide a probabilistic model that can be used in machine learning tasks. More recently , there there are a number of methods that have been introduced that capture manifold structure through a variety of approaches that involve estimating local tangent planes BID9 a; BID3 BID26 . While these methods can admit representations that have some of the above advantages of generative models, the linear approximations can cause challenges when trying to perform transfer learning by extrapolating to out-of-sample points in manifold locations not well-represented in the training data.As an alternative to the above, previous work has proposed unsupervised learning algorithms to efficiently learn Lie group operators that capture the structure of low-dimensional manifolds BID6 BID33 BID24 BID28 . The manifold representation that is described by analytic operators (called transport operators) defines the process of transporting one data point to another, thereby providing a probabilistic generative model for the manifold structure BID6 ). The main contribution of this paper is to define two transfer learning methods that use this generative manifold representation to learn natural transformations and incorporate them into new data. The first method uses this representation in a novel randomized approach to transfer learning that employs the learned generative model to map out unseen regions of the data space. These results are shown through demonstrations of transfer learning in a data augmentation task for few-shot image classification. The second method use of transport operators for injecting specific transformations into new data examples which allows for realistic image animation and informed data augmentation. We have shown that we can transfer meaninful transformation information between classes and examples using manifold transport operators which provide accurate and robust characterization of manifold data in the context of a probabilistic generative model. We have demonstrated that the learned transformations can be transferred accurately to other portions of the manifold (including out-of-sample extensions) through applying the generative model with both randomized and structured coefficients. The transfer learning potential was shown in the context of data generation and augmentation applications by animating individuals with new facial expression and providing examples of few-shot learning on digit and facial expression classification. These results constitute some Classification Accuracy Figure 9 : The boxes represent the classification accuracy distribution when the classifier is trained using the transport operator-augmented dataset (transport operator), using only one example per expression without data augmentation (single example), and using landmarks from the expression frames in all of the training sequences from the CK+ database (CK+).of the first demonstrations that transport operators can form the basis of a learned representation for manifold data that has utility in applications and transfer learning tasks. The presented simulations are proof-of-concept simulations that allow us to fully explore and visualize the results in a way that is impossible with larger or more complex datasets.While successful in these demonstrations, as with any algorithm the results depend on the data characteristics being sufficiently captured by the model family. Though there is evidence in the literature that Lie group operators can capture complex transformations in images BID6 , it is unknown if the transport operator approach will be sufficient to represent the rich variations in more complex datasets. Future work will be required to determine the complexity of manifold models that can be captured successfully by the proposed manifold transport operator approach, including demonstrations of similar tasks on more complex image classification tasks. Additionally , the current model only captures local transformations between points, but it will likely be beneficial to develop the model further to capture more regularity in the coefficient behavior across multiple pairs of points.
Learning transport operators on manifolds forms a valuable representation for doing tasks like transfer learning. Uses a dictionary learning framework to learn manifold transport operators on augmented USPS digits. The paper considers the framework of manifold transport operator learning of Culpepper and Olshausen (2009), and interprets it as obtaining a MAP estimate under a probabilistic generative model.