Frequency-aware Dimension Selection for Static Word Embedding by Mixed
  Product Distance
Lingfeng Shen Haiyun Jiang Lemao Liu Ying Chen
Static word embedding is still useful, particularly for context-unavailable
tasks, because in the case of no context available, pre-trained language models
often perform worse than static word embeddings. Although dimension is a key
factor determining the quality of static word embeddings, automatic dimension
selection is rarely discussed. In this paper, we investigate the impact of word
frequency on the dimension selection, and empirically find that word frequency
is so vital that it needs to be taken into account during dimension selection.
Based on such an empirical finding, this paper proposes a dimension selection
method that uses a metric (Mixed Product Distance, MPD) to select a proper
dimension for word embedding algorithms without training any word embedding.
Through applying a post-processing function to oracle matrices, the MPD-based
method can de-emphasize the impact of word frequency. Experiments on both
context-unavailable and context-available tasks demonstrate the better
efficiency-performance trade-off of our MPD-based dimension selection method
over baselines.