A Bias-Variance-Covariance Decomposition of Kernel Scores for Generative
  Models
Sebastian G. Gruber Florian Buettner
Generative models, like large language models, are becoming increasingly
relevant in our daily lives, yet a theoretical framework to assess their
generalization behavior and uncertainty does not exist. Particularly, the
problem of uncertainty estimation is commonly solved in an ad-hoc manner and
task dependent. For example, natural language approaches cannot be transferred
to image generation. In this paper we introduce the first
bias-variance-covariance decomposition for kernel scores and their associated
entropy. We propose unbiased and consistent estimators for each quantity which
only require generated samples but not the underlying model itself. As an
application, we offer a generalization evaluation of diffusion models and
discover how mode collapse of minority groups is a contrary phenomenon to
overfitting. Further, we demonstrate that variance and predictive kernel
entropy are viable measures of uncertainty for image, audio, and language
generation. Specifically, our approach for uncertainty estimation is more
predictive of performance on CoQA and TriviaQA question answering datasets than
existing baselines and can also be applied to closed-source models.