Media
VideoChecking Robustness in 4 Steps
Michèle B. NuijtenWhether a published finding is robust is difficult to assess. Researchers often point at replication as a robustness check. However, conducting a replication on a new sample can cost a lot of time, effort, and money. In this talk, I propose a consecutive “four-step robustness check” that aims at the low-hanging fruit first. First, we check the internal consistency of statistical results (possibly using automated tools, such as “statcheck”).
Second, we reanalyze the data using the original analytical strategy to see if the reported conclusions hold. Third, we check if the original result is robust to alternative analytical choices, for instance via a multi-verse analysis. Only then, in the fourth step, we perform a replication study on a new sample. This four-step approach allows detecting unreliable results, while wasting as little resources as possible. I will discuss potential advantages and limitations of this approach.