A/B TESTING
STATISTICAL GUIDE.

Core Concepts

• Null Hypothesis (H₀) – No difference between variants.
• Alternative Hypothesis (H₁) – There is a difference (one‑ or two‑sided).
• Test Statistic – Calculated from data to compare to a distribution.
• p‑value – Probability of observing data as extreme as observed if H₀ is true.
• Statistical Significance – Usually p < 0.05 (α = 0.05).
• Confidence Interval – Range containing the true effect with given confidence.
• Power (1‑β) – Probability of detecting an effect if it exists.
• Effect Size – Magnitude of difference (e.g., relative lift).

---

Hypothesis Testing Steps

1. Define the problem – what metric to improve?
2. Formulate hypotheses – H₀ and H₁.
3. Choose test type – one‑sided or two‑sided.
4. Determine sample size – based on desired power, α, effect size.
5. Randomise assignment – users to control or treatment.
6. Run experiment – collect data.
7. Check assumptions – normality, independence, etc.
8. Compute test statistic and p‑value.
9. Interpret results – reject or fail to reject H₀.
10. Act on findings – implement treatment if significant.

---

Common Statistical Tests

Scenario	Test	Python / R Function
Two independent means (normal)	t‑test (Student's)	scipy.stats.ttest_ind
Two independent proportions	z‑test (or chi‑square)	statsmodels.stats.proportion.proportions_ztest
Paired comparison (pre/post)	Paired t‑test	scipy.stats.ttest_rel
More than two groups	ANOVA	scipy.stats.f_oneway
Non‑parametric (two groups)	Mann‑Whitney U	scipy.stats.mannwhitneyu
Categorical association	Chi‑square test	scipy.stats.chi2_contingency

---

Sample Size Calculation

For Continuous Metrics (e.g., conversion rate)

# Using statsmodels
from statsmodels.stats.power import NormalIndPower
from statsmodels.stats.proportion import proportion_effectsize

# For proportion (e.g., conversion rate)
effect_size = proportion_effectsize(prop1=0.10, prop2=0.12)  # 10% → 12%
power_analysis = NormalIndPower()
sample_size = power_analysis.solve_power(
    effect_size=effect_size,
    power=0.80,        # desired power
    alpha=0.05,        # significance level
    ratio=1.0,         # 1:1 allocation
    alternative='two-sided'
)
print(f'Sample size per group: {sample_size:.0f}')

# For continuous (e.g., revenue per user)
# Use TTestIndPower
from statsmodels.stats.power import TTestIndPower
effect_size = 0.2  # Cohen's d (small effect)
power_analysis = TTestIndPower()
n = power_analysis.solve_power(effect_size=effect_size, power=0.8, alpha=0.05, ratio=1.0)
print(f'Sample size per group: {n:.0f}')

---

Interpreting Results

p‑value Interpretation

• p < 0.05 – reject H₀; statistically significant difference.
• p ≥ 0.05 – fail to reject H₀; insufficient evidence.
• Caution – p‑value is not the probability that H₀ is true.

Confidence Intervals

• If the 95% CI for the difference does not include 0, the result is significant.
• If it includes 0, not significant.
• Provides range of plausible effect sizes.

Practical Significance

• Statistically significant ≠ practically significant.
• Consider business impact (e.g., 0.1% lift on conversion may not be worth the effort).
• Always assess effect size alongside p‑value.

---

Common Pitfalls

• Peeking – checking results early and stopping (inflates false positive rate).
• Multiple testing – running many tests increases chance of false positives (apply Bonferroni correction).
• Sample size too small – lack of power to detect meaningful effect.
• Non‑random assignment – selection bias.
• Novelty effect – initial reaction to change may fade.
• Seasonality – results may vary over time.
• Network effects – user interactions can violate independence (e.g., social networks).
• Simpson's paradox – aggregated results differ from subgroups.

---

Advanced Topics

Sequential Testing

• Use methods that allow early stopping with controlled error rates.
• Example: Pocock, O'Brien‑Fleming boundaries.
• Implemented in statsmodels.stats.multitest or dedicated packages.

Bayesian A/B Testing

• Instead of p‑values, compute probability that variant B is better.
• Use posterior distributions (e.g., Beta‑Binomial).
• Can incorporate prior information.
• Example: pymc or bayesian‑ab package.

---

Python Implementation Example (Frequentist)

import numpy as np
from scipy import stats
from statsmodels.stats.proportion import proportions_ztest
import pandas as pd

# Simulate data
np.random.seed(42)
n_control = 1000
n_treatment = 1000
p_control = 0.10
p_treatment = 0.12

control = np.random.binomial(1, p_control, n_control)
treatment = np.random.binomial(1, p_treatment, n_treatment)

# Convert to counts
conversions_control = control.sum()
conversions_treatment = treatment.sum()
n_control = len(control)
n_treatment = len(treatment)

# z‑test for proportions
z_stat, p_value = proportions_ztest(
    [conversions_control, conversions_treatment],
    [n_control, n_treatment],
    alternative='two-sided'
)

# Confidence interval
from statsmodels.stats.proportion import proportion_confint
ci_control = proportion_confint(conversions_control, n_control)
ci_treatment = proportion_confint(conversions_treatment, n_treatment)

print(f'Control CR: {conversions_control/n_control:.3f} (95% CI: {ci_control[0]:.3f} - {ci_control[1]:.3f})')
print(f'Treatment CR: {conversions_treatment/n_treatment:.3f} (95% CI: {ci_treatment[0]:.3f} - {ci_treatment[1]:.3f})')
print(f'z-statistic: {z_stat:.3f}, p-value: {p_value:.4f}')

if p_value < 0.05:
    print('Reject H₀ – significant difference')
else:
    print('Fail to reject H₀ – not significant')

---

Bayesian A/B Testing Example (Beta‑Binomial)

import numpy as np
from scipy.stats import beta

# Prior: Beta(1,1) = Uniform
alpha_prior = 1
beta_prior = 1

# Convert observed data
alpha_control = alpha_prior + conversions_control
beta_control = beta_prior + n_control - conversions_control
alpha_treatment = alpha_prior + conversions_treatment
beta_treatment = beta_prior + n_treatment - conversions_treatment

# Sample from posterior
samples_control = beta.rvs(alpha_control, beta_control, size=100000)
samples_treatment = beta.rvs(alpha_treatment, beta_treatment, size=100000)

# Probability treatment is better
prob_treatment_better = np.mean(samples_treatment > samples_control)
print(f'P(Treatment > Control) = {prob_treatment_better:.3f}')

# Expected lift
expected_lift = np.mean((samples_treatment - samples_control) / samples_control)
print(f'Expected relative lift = {expected_lift:.2%}')

# Credible interval for difference
diff = samples_treatment - samples_control
ci_low, ci_high = np.percentile(diff, [2.5, 97.5])
print(f'95% Credible interval for difference: [{ci_low:.3f}, {ci_high:.3f}]')

---

Metrics and KPIs

• Conversion Rate – binary outcome (purchased, signed up, etc.).
• Revenue per User – continuous metric.
• Retention / Churn – longitudinal metrics.
• Click‑Through Rate (CTR) – for ads or links.
• Engagement – time spent, pages viewed, etc.
• Net Promoter Score (NPS) – ordinal survey response.

---

Best Practices

• Pre‑register – document hypothesis, metrics, sample size before running.
• Randomise properly – use deterministic hashing or a true random generator.
• Use a holdout group – if running multiple experiments simultaneously.
• Monitor in real‑time – watch for anomalies (e.g., data quality issues).
• Run for sufficient duration – account for weekly cycles.
• Segment results – check for heterogeneous effects (e.g., new vs. returning users).
• Apply corrections – for multiple comparisons if testing many variants.
• Use Bayesian approaches – for more intuitive interpretation and early stopping.
• Automate – use statistical platforms or internal tools to standardise.
• Document decisions – record what was tested, results, and actions taken.

---

Tools and Libraries

• Python – scipy.stats, statsmodels, pymc (Bayesian).
• R – built‑in functions (t.test, prop.test), pwr package for power.
• Online calculators – Evan Miller's A/B test calculator.
• Optimizely / VWO – commercial platforms with built‑in analysis.

---

Quick Reference Table of Tests

Metric Type	Design	Test	Assumptions
Binary (conversion)	Two independent groups	z‑test (proportions)	Large sample, independent observations
Continuous (revenue)	Two independent groups	t‑test (Welch's)	Approx. normal or large sample
Continuous (pre‑post)	Paired	Paired t‑test	Differences approx. normal
Binary (pre‑post)	Paired	McNemar's test	Paired binary outcomes
Ordinal (NPS)	Two groups	Mann‑Whitney U	Non‑parametric, ordinal
Multiple variants	Independent	ANOVA (continuous) or Chi‑square (categorical)	Normal / independence