6 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
The article outlines six indicators that suggest an experiment should be repeated, such as solid impact results, almost significant p-values, and cases where initial results seem "too good to be true." It emphasizes the importance of revisiting past experiments for better insights and improving statistical power.
If you do, here's more
Old experiments often hold untapped potential. Rerunning them can uncover effects that were previously obscured due to limited statistical power or sample size. The piece identifies six clear signals indicating when to consider a rerun. First, experiments with solid impact should be revisited, especially if they can reach a broader audience or apply more intense treatment variables. Second, results that were nearly significant—p-values between 0.01 and 0.1—merit another look. These can benefit from increased statistical power and meta-analysis.
The third signal is when results appear too good to be true, marked by large effects from small samples or unexpected outcomes. Such cases may warrant skepticism and re-examination. Conversely, if an experiment shows no impact despite strong supporting metrics, it could indicate a false negative, suggesting a rerun might be useful. The fifth signal points to interesting segments; if A/B tests yield average results but show promise in smaller groups, that’s another reason to revisit the experiment. Finally, when multiple teams report similar insignificant results, a meta-analysis could reveal clearer insights and justify another attempt.
The author expresses interest in collaborating with teams who have a substantial history of experiments. They hint at developing a tool to assist in identifying candidates for reruns. The mention of “strong losers” being inverted with new treatments adds an interesting layer, suggesting that past failures can sometimes lead to future successes with adjustments. This perspective encourages a proactive approach to experimentation, emphasizing the need to revisit and refine past efforts rather than dismissing them outright.
Questions about this article
No questions yet.