Embeddings are a fundamental component of many modern machine learning and natural language processing models.
 Understanding them and visualizing them is essential for gathering insights about the information they capture and the behavior of the models.
 State of the art in analyzing embeddings consists in projecting them in two-dimensional planes without any interpretable semantics associated to the axes of the projection, which makes detailed analyses and comparison among multiple sets of embeddings challenging.
 In this work, we propose to use explicit axes defined as algebraic formulae over embeddings to project them into a lower dimensional, but semantically meaningful subspace, as a simple yet effective analysis and visualization methodology.
 This methodology assigns an interpretable semantics to the measures of variability and the axes of visualizations, allowing for both comparisons among different sets of embeddings and fine-grained inspection of the embedding spaces.
 We demonstrate the power of the proposed methodology through a series of case studies that make use of visualizations constructed around the underlying methodology and through a user study. The results show how the methodology is effective at providing more profound insights than classical projection methods and how it is widely applicable to many other use cases. Learning representations is an important part of modern machine learning and natural language processing research. Those representations are often real-valued vectors also called embeddings and are obtained both as byproducts of supervised learning or as the direct goal of unsupervised methods. Independently of how the embeddings are learned, there is much value in understanding what information they capture, how they relate to each other and how the data they are learned from influences them. A better understanding of the embedded space may lead to a better understanding of the data, of the problem and the behavior of the model, and may lead to critical insights in improving such models. Because of their high-dimensional nature, they are hard to visualize effectively, and the most adopted approach is to project them in a bi-dimensional space. Projections have a few shortcomings: 1) they may not preserve distance in the original space, 2) they are not comparable across models and 3) do not provide interpretable dimensions of variability to project to, preventing for more detailed analysis and understanding. For these reasons, there is value in mapping embeddings into a more specific, controllable and interpretable semantic space.Principal Component Analysis (PCA) BID27 and t-Distributed Stochastic Neighbor Embedding (t-SNE) BID30 are two projection techniques often used for visualizing embeddings in two dimensions, although other techniques can be used. PCA projects embeddings on a lower dimensional space that has the directions of the highest variance in the dataset as axes. Those dimensions do not carry any interpretable meaning, making interpretation difficult. By visualizing the first two dimensions of a PCA projection, the only insight obtainable is semantic relatedness BID5 between points by observing their relative closeness and therefore topical clusters can be identified. The downside is that embeddings that end up being close in the projected space may not be close in the original embedding space and vice versa. Moreover, as the directions of highest variance are different from embedding space to embedding space, the projections are incompatible among different embeddings spaces, and this makes them not comparable, a common issue among dimensionality reduction techniques. t-SNE, differently from PCA, optimizes a loss that encourages embeddings that are close in the original high-dimensional space to be close in the lower dimensional projection space. This helps in visualizing clusters better than with PCA, as t-SNE puts each point in the projected space so that distance in the original space with respect to its nearest neighbors is preserved as much as possible. Visualizations obtained in this way reflect more the original embedding space and topical clusters are more clearly distinguishable, but doesn't solve the issue of comparability of two different sets of embeddings, nor it solves the lack of interpretability of the axes and still doesn't allow for finegrained inspection. Moreover, t-SNE is pretty sensible to hyperparameters, making it unclear how much the projection reflects the data.In this paper, a new and simple method to inspect, explore and debug embedding spaces at a finegrained level is proposed. It consists in defining explicitly the axes of projection through formulae in vector algebra over the embeddings themselves. Explicit axis definition gives an interpretable and fine-grained semantics to the axes of projection. Defining axes explicitly makes it possible to analyze in a detailed way how embeddings relate to each other with respect to interpretable dimensions of variability, as carefully crafted formulas can map (to a certain extent) to semantically meaningful portions of the learned spaces. The explicit axes definition also allows for comparing of embeddings obtained from different datasets, as long as they have common labels.We demonstrate three visualizations for analyzing subspaces of interest of embedding spaces and a set of example case studies including bias detection, polysemy analysis and fine-grained embedding analysis. Additional tasks that may be performed using the proposed methodology and visualization are diachronic analysis and analysis of representations learned from graphs and knowledge bases. The proposed visualizations can moreover be used for debugging purposes and in general for obtaining a better understanding of the embedding spaces learned by different models and representation learning approaches. We are releasing an open-source 1 interactive tool that implements the proposed visualizations, in order to enable researchers in the fields of machine learning, computational linguistics, natural language processing, social sciences and digital humanities to perform exploratory analysis and better understand the semantics of their embeddings.The main contribution of this work lies in the use of explicit user-defined algebraic formulae as axes for projecting embedding spaces into semantically-meaningful subspaces that when visualized provide interpretable axes. We show how this methodology can be widely used through a series of case studies on well known models and data and we furthermore validate the how the visualizations are more interpretable through a user study. We presented a simple methodology for projecting embeddings into lower-dimensional semantically-meaningful subspaces through explicit vector algebra formulae operating on the embedding themselves. Classical projection methods are useful to gather on overall coarse-grained view of the embedding space and how embeddings cluster, but we showed how our approach allows goal-oriented analyses with more fine-grained comparison and enables cross-dataset comparison through a series of case studies and a user study. This is possible thanks to the ability of the proposed methodology to assign an explicit semantics to the measures of variability used as axes of the visualization that in turns makes them interpretable and widely applicable to many use cases in computational linguistics, natural language processing, machine learning, social sciences and digital humanities.A APPENDIX Figure 6 : Professions plotted on "male" and "female" axes in W ikipedia embeddings.
We propose to use explicit vector algebraic formulae projection as an alternative way to visualize embedding spaces specifically tailored for goal-oriented analysis tasks and it outperforms t-SNE in our user study. Analysis of embedding psaces in a non-parametric (example-based_ way