Optional stopping and the false discovery rate

less than 1 minute read

An undergraduate training in statistics tells us that sequential testing with optional stopping inflates the false positive rate, but exactly how much it inflates it is less intuitive. Sometimes, the metaphor of rolling a 20-sided dice every time a statistical test is run (assuming alpha = 0.05). However, whereas each roll of the dice is independent, running a new instance of a test on a slightly data set after adding a number of participants is not fully independent. As such, 20-retests after adding each of 20 new participants is unlikely to inflate the false-positive rate all the way to 1.0. Sequential testing with optional stopping is bad, no doubt, but just how bad? I had to run a simulation study to figure it out. You can find it here.