Evaluating Perceptual Distances by Fitting Binomial Distributions to
  Two-Alternative Forced Choice Data
Alexander Hepburn Raul Santos-Rodriguez Javier Portilla
The two-alternative forced choice (2AFC) experimental setup is popular in the
visual perception literature, where practitioners aim to understand how human
observers perceive distances within triplets that consist of a reference image
and two distorted versions of that image. In the past, this had been conducted
in controlled environments, with a tournament-style algorithm dictating which
images are shown to each participant to rank the distorted images. Recently,
crowd-sourced perceptual datasets have emerged, with no images shared between
triplets, making ranking impossible. Evaluating perceptual distances using this
data is non-trivial, relying on reducing the collection of judgements on a
triplet to a binary decision -- which is suboptimal and prone to misleading
conclusions. Instead, we statistically model the underlying decision-making
process during 2AFC experiments using a binomial distribution. We use maximum
likelihood estimation to fit a distribution to the perceptual judgements,
conditioned on the perceptual distance to test and impose consistency and
smoothness between our empirical estimates of the density. This way, we can
evaluate a different number of judgements per triplet, and can calculate
metrics such as likelihoods of judgements according to a set of distances --
key ingredients that neural network counterparts lack.