Assessing effect sizes, variability, and power in the on-line study of
  language production
Bürki Audrey Vasishth Shravan
With the pandemic, many experimental psychologists and linguists have started
to collect data over the internet (hereafter on-line data). The feasibility of
such experiments and the sample sizes required to achieve sufficient
statistical power in future experiments have to be assessed. This in turn
requires information on effect sizes and variability. In a series of analyses,
we compare response time data obtained in the same word production experiment
conducted in the lab and on-line. These analyses allow us to determine whether
the two settings differ in effect sizes, in the consistency of responses over
the course of the experiment, in the variability of average response times
across participants, in the magnitude of effect sizes across participants, or
in the amount of unexplained variability. We assess the impact of these
differences on the power of the design in a series of simulations. Our findings
temper the enthusiasm raised by previous studies and suggest that on-line
production studies might be feasible but at a non-negligible cost. The sample
sizes required to achieve sufficient power in on-line language production
studies come with a non-negligible increase in the amount of manual labour.