LiDAR: Sensing Linear Probing Performance in Joint Embedding SSL
  Architectures
Vimal Thilak Chen Huang Omid Saremi Laurent Dinh Hanlin Goh Preetum Nakkiran Joshua M. Susskind Etai Littwin
Joint embedding (JE) architectures have emerged as a promising avenue for
acquiring transferable data representations. A key obstacle to using JE
methods, however, is the inherent challenge of evaluating learned
representations without access to a downstream task, and an annotated dataset.
Without efficient and reliable evaluation, it is difficult to iterate on
architectural and training choices for JE methods. In this paper, we introduce
LiDAR (Linear Discriminant Analysis Rank), a metric designed to measure the
quality of representations within JE architectures. Our metric addresses
several shortcomings of recent approaches based on feature covariance rank by
discriminating between informative and uninformative features. In essence,
LiDAR quantifies the rank of the Linear Discriminant Analysis (LDA) matrix
associated with the surrogate SSL task -- a measure that intuitively captures
the information content as it pertains to solving the SSL task. We empirically
demonstrate that LiDAR significantly surpasses naive rank based approaches in
its predictive power of optimal hyperparameters. Our proposed criterion
presents a more robust and intuitive means of assessing the quality of
representations within JE architectures, which we hope facilitates broader
adoption of these powerful techniques in various domains.