Is Cosine-Similarity of Embeddings Really About Similarity?
Harald Steck Chaitanya Ekanadham Nathan Kallus
Cosine-similarity is the cosine of the angle between two vectors, or
equivalently the dot product between their normalizations. A popular
application is to quantify semantic similarity between high-dimensional objects
by applying cosine-similarity to a learned low-dimensional feature embedding.
This can work better but sometimes also worse than the unnormalized dot-product
between embedded vectors in practice. To gain insight into this empirical
observation, we study embeddings derived from regularized linear models, where
closed-form solutions facilitate analytical insights. We derive analytically
how cosine-similarity can yield arbitrary and therefore meaningless
`similarities.' For some linear models the similarities are not even unique,
while for others they are implicitly controlled by the regularization. We
discuss implications beyond linear models: a combination of different
regularizations are employed when learning deep models; these have implicit and
unintended effects when taking cosine-similarities of the resulting embeddings,
rendering results opaque and possibly arbitrary. Based on these insights, we
caution against blindly using cosine-similarity and outline alternatives.