Uncovering Meanings of Embeddings via Partial Orthogonality
Yibo Jiang Bryon Aragam Victor Veitch
Machine learning tools often rely on embedding text as vectors of real
numbers. In this paper, we study how the semantic structure of language is
encoded in the algebraic structure of such embeddings. Specifically, we look at
a notion of ``semantic independence'' capturing the idea that, e.g.,
``eggplant'' and ``tomato'' are independent given ``vegetable''. Although such
examples are intuitive, it is difficult to formalize such a notion of semantic
independence. The key observation here is that any sensible formalization
should obey a set of so-called independence axioms, and thus any algebraic
encoding of this structure should also obey these axioms. This leads us
naturally to use partial orthogonality as the relevant algebraic structure. We
develop theory and methods that allow us to demonstrate that partial
orthogonality does indeed capture semantic independence. Complementary to this,
we also introduce the concept of independence preserving embeddings where
embeddings preserve the conditional independence structures of a distribution,
and we prove the existence of such embeddings and approximations to them.