Bayesianbuddy (BETA)
powered by Experiment Nation
If you work in Conversion Rate Optimization (CRO) or Product Experimentation, sometimes a Frequentist approach just isn’t the right fit — maybe your sample size is small, and you simply want a reasonably good chance of identifying a winner. In that case, a Bayesian approach could be a better fit for you.
Based on facing this problem at several companies, I created a simple Bayesian Calculator to help you figure out test winners, probability of winning, and risk of promoting a losing variant so you can make more informed decisions.
You can share results with others, compare multiple variants, set your own thresholds for risk, as well as success.
How Do you use BayesianBuddy?
You will need a few parameters to start:
Historical Conversion Rate (something representative of your data); And historical standard deviation for continuous metrics.
Risk Threshold (what relative percentage loss are you OK with if you promote the wrong variant)
Threshold of Caring (what is the minimum relative percentage lift you care about)
Impressions of each variant
Conversions of each variant; And standard deviation for continuous metrics.
BayesianBuddy will then output for each variant
Probability of winning
Associated risk
Relative Lift
Privacy Notes
You can share a link to the results for review, but the original uploaded data will no longer be available in BayesianBuddy.