Are data collected on the internet more variable?

With the pandemic, many experimental psychologists/linguists have started to collect data over the internet (“on-line data”). It is often assumed that data collected over the internet are more variable than lab data.

Is this true? If so, what is the source of this variability?

• Are data more variable because participants differ more from one another?
• Are data more variable because participants are more distracted or because many things are going on at once on their computer?

Knowledge about the extent and sources of variability is necessary to determine the feasibility of such experiments and compute the sample sizes required to achieve sufficient statistical power in future experiments.

We collected data in the lab and on-line for the same experiment. We measured the time needed to start saying the name of an object when this object is shown on the screen. The graph below suggests that on-line data are more variable:

Analyses show:

• That on-line data are more variable
• That the variability mostly stems from differences between participants

So, do we need larger sample sizes than for a lab experiment?

• For between-participant designs, for sure

• For within participant designs, not necessarily

• The graphs below show power as a function of number of participants for two effects of typical size, 90 items per condition.