Full text PDF

Researchers sometimes make two flexible choices during a study: they look at results before data collection is finished, and they try several defensible analyses before deciding what to report. We call the joint use of these practices Forking-Peeks. Using computer simulations of simple two-group studies, we show that either practice alone can increase false-positive findings, but their combination can do so dramatically. In one illustrative setting with 16 correlated analysis options and four interim looks at the accumulating data, a nominal 5% false-positive rate rose to about 70% when the analyst repeatedly selected the best-looking analysis. Even a seemingly more consistent strategy, choosing the best-looking analysis once and then keeping it fixed, still produced severe inflation because the initial choice was data-dependent. The same selection process also made statistically significant effects look larger than they really were. The manuscript provides reproducible code and visual risk maps to help readers understand how small, plausible research decisions can compound when data collection and analysis remain flexible. We conclude with practical recommendations: plan stopping rules in advance, use valid sequential designs when interim looks are needed, treat multiple analyses as multiplicity, and report effect sizes and intervals rather than relying on a single threshold.

Pages © TQMP;
Website last modified: 2026-09-20.
Template last modified: 2022-03-04 18h27.
Page consulted on .
Be informed of the upcoming issues with RSS feed: RSS