Estimating Unknown Population Sizes Using the Hypergeometric
  Distribution
Liam Hodgson Danilo Bzdok
The multivariate hypergeometric distribution describes sampling without
replacement from a discrete population of elements divided into multiple
categories. Addressing a gap in the literature, we tackle the challenge of
estimating discrete distributions when both the total population size and the
sizes of its constituent categories are unknown. Here, we propose a novel
solution using the hypergeometric likelihood to solve this estimation
challenge, even in the presence of severe under-sampling. We develop our
approach to account for a data generating process where the ground-truth is a
mixture of distributions conditional on a continuous latent variable, such as
with collaborative filtering, using the variational autoencoder framework.
Empirical data simulation demonstrates that our method outperforms other
likelihood functions used to model count data, both in terms of accuracy of
population size estimate and in its ability to learn an informative latent
space. We demonstrate our method's versatility through applications in NLP, by
inferring and estimating the complexity of latent vocabularies in text
excerpts, and in biology, by accurately recovering the true number of gene
transcripts from sparse single-cell genomics data.