2 min read
|
Saved February 14, 2026
|
Copied!
Do you care about this?
The article argues that not all good ideas can be validated through controlled experiments. It highlights the risks of discarding potentially valuable changes and suggests using a broader approach to validation, emphasizing the importance of learning from all experiments, regardless of their outcomes.
If you do, here's more
The article emphasizes that failing to show statistically significant results in controlled experiments doesnβt negate the potential value of an idea. It argues that if you only pursue tests labeled as "winners," you risk discarding a significant amount of possibly valuable insights. For instance, research from the site GuessTheTest indicates that people guess correctly about 60% of the time regarding the success of ideas, hinting that out of ten ideas you consider good, six may actually hold merit. Stefan Thomke's study, which analyzed over 22,000 experiments across 1,300 websites, reveals that only 10% of those experiments demonstrated statistically significant results.
The author suggests a shift in mindset toward experimentation. Recognizing that controlled tests are just one method of validation is key. When a test fails to meet statistical standards, other forms of validation can be more effective than improperly conducted tests. Risk management plays a central role in decision-making. Rather than focusing solely on "winning" tests, every experiment should be viewed as an opportunity for learning, even if the results arenβt significant. This perspective encourages teams to avoid discarding potentially good ideas based on a narrow interpretation of success.
Contributions from the comments add depth to the discussion. One commenter highlights the effort involved in running A/B tests, noting that companies delivering statistically significant results often face longer time-to-market due to high internal demands. Another emphasizes the need for a blended research approach, combining usability testing and surveys with A/B tests to enhance decision-making and speed up development. The conversation reflects a collective recognition that while significant wins are infrequent, informed decisions supported by various data points can still be valuable.
Questions about this article
No questions yet.