Embracing Variation and Accepting Uncertainty: Implications for Science and MetascienceAndrew Gelman Related
The world would be pretty horrible if your attitude on immigration could be affected by a subliminal smiley face, if elections were swung by shark attacks and college football games, if how you vote depended on the day within your monthly cycle, etc. Fortunately, there is no good evidence for these and other high-profile claims about the effects of apparently irrelevant stimuli on social and political attitudes and behaviors.
Indeed, for theoretical reasons, we argue that it is not possible for these large and persistent effects to co-exist in the real world. But if the sorts of effects being studied vary greatly by person and scenario, then simple experiments will not yield reliable estimates of effect sizes. It is necessary to instead embrace variation, which, in turn, requires accepting uncertainty. This has implications for the practice of science and for the proper understanding of replication and other aspects of metascience.