Statistics for Business — Hypothesis Testing and A/B Tests¶
Hypothesis testing is how analysts turn a business question into a yes/no answer with a quantified level of confidence. A/B testing is the gold standard for measuring the impact of product changes. This file covers what analysts actually need to know — the intuition, the practice, and the traps.
Learning Objectives¶
- Understand the logic of hypothesis testing (null vs alternative)
- Interpret p-values correctly
- Construct and interpret confidence intervals
- Design and analyse a simple A/B test
- Recognise common statistical mistakes
Hypothesis Testing — The Logic¶
A hypothesis test answers: "Is this result likely to occur by chance, or is there a real effect?"
Steps: 1. H₀ (Null hypothesis): "There is no effect — any observed difference is random chance" 2. H₁ (Alternative hypothesis): "There is a real effect" 3. Set significance level α — the threshold for rejecting H₀ (typically 0.05) 4. Compute a test statistic — depends on the test 5. Get the p-value — probability of observing this result if H₀ is true 6. Decision: if p < α, reject H₀ (we have evidence for H₁)
The P-Value — What It Actually Means¶
The p-value is the probability of observing a result this extreme (or more extreme), assuming the null hypothesis is true.
Common misinterpretations: - ❌ "p = 0.03 means there's a 3% chance the null hypothesis is true" - ❌ "p = 0.03 means we're 97% confident the effect is real" - ✓ "If H₀ were true, we'd only see a difference this large 3% of the time by chance"
The T-Test — Comparing Two Means¶
One-sample t-test: Does the mean equal a specific value?
from scipy import stats
import numpy as np
# Q: Is the average order value significantly different from £200?
orders = [189, 245, 298, 165, 310, 189, 420, 155, 280, 190]
t_stat, p_value = stats.ttest_1samp(orders, popmean=200)
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")
print(f"Reject H₀ at 5%: {p_value < 0.05}")
Two-sample t-test: Are two groups different?
import pandas as pd
df = pd.read_csv("orders_clean.csv")
df["total"] = df["quantity"] * df["unit_price"]
electronics = df[df["category"] == "Electronics"]["total"]
books = df[df["category"] == "Books"]["total"]
# Welch's t-test (doesn't assume equal variance)
t_stat, p_value = stats.ttest_ind(electronics, books, equal_var=False)
print(f"Electronics mean: £{electronics.mean():.2f}")
print(f"Books mean: £{books.mean():.2f}")
print(f"Difference: £{electronics.mean() - books.mean():.2f}")
print(f"t-statistic: {t_stat:.3f}")
print(f"p-value: {p_value:.4f}")
print(f"Significant at 5%: {p_value < 0.05}")
Confidence Intervals¶
A 95% confidence interval means: "If we repeated this experiment many times, 95% of the intervals we construct would contain the true population mean."
import numpy as np
from scipy import stats
data = df["total"].dropna()
mean = data.mean()
sem = stats.sem(data) # standard error of the mean
ci = stats.t.interval(0.95, df=len(data)-1, loc=mean, scale=sem)
print(f"Mean: £{mean:.2f}")
print(f"95% CI: £{ci[0]:.2f} to £{ci[1]:.2f}")
Interpretation: We're 95% confident the true average order value in the population falls between £[lower] and £[upper].
Common misinterpretation
A 95% CI does NOT mean "there's a 95% probability the true mean is in this interval." Once computed, the true mean either is or isn't in it. The 95% refers to the long-run success rate of the method.
A/B Testing — The Analyst's Most Important Skill¶
An A/B test is a controlled experiment: you randomly assign users to two groups (A = control, B = treatment), measure an outcome, and use statistics to determine if B is better.
Step 1 — Define the hypothesis and metric¶
# Business question: Does a red "Buy Now" button (B)
# convert better than the current blue one (A)?
# H₀: conversion_B = conversion_A (no difference)
# H₁: conversion_B > conversion_A (B converts better)
# Metric: conversion rate (% of visitors who purchase)
# α = 0.05 (5% significance level)
Step 2 — Calculate required sample size¶
from statsmodels.stats.power import NormalIndPower
# Minimum detectable effect: we want to detect a 2% absolute improvement
baseline_rate = 0.12 # current 12% conversion
min_effect = 0.02 # detect if it goes to 14%
# Rule of thumb: use power analysis
analysis = NormalIndPower()
sample_size = analysis.solve_power(
effect_size=min_effect / (baseline_rate * (1 - baseline_rate)) ** 0.5,
power=0.80, # 80% chance of detecting a real effect
alpha=0.05, # 5% false positive rate
ratio=1.0 # equal group sizes
)
print(f"Required sample size per group: {int(sample_size):,}")
Step 3 — Run the test¶
# Simulated results after collecting data
group_a = {"visitors": 1500, "conversions": 183} # 12.2%
group_b = {"visitors": 1500, "conversions": 225} # 15.0%
rate_a = group_a["conversions"] / group_a["visitors"]
rate_b = group_b["conversions"] / group_b["visitors"]
print(f"Control (A): {rate_a:.1%} conversion")
print(f"Treatment (B): {rate_b:.1%} conversion")
print(f"Absolute lift: {rate_b - rate_a:.1%}")
print(f"Relative lift: {(rate_b - rate_a) / rate_a:.1%}")
Step 4 — Test for significance¶
from scipy.stats import chi2_contingency
import numpy as np
# Chi-square test for proportions
contingency_table = np.array([
[group_a["conversions"], group_a["visitors"] - group_a["conversions"]],
[group_b["conversions"], group_b["visitors"] - group_b["conversions"]],
])
chi2, p_value, dof, expected = chi2_contingency(contingency_table)
print(f"\nChi-square: {chi2:.3f}")
print(f"p-value: {p_value:.4f}")
print(f"Significant at 5%: {p_value < 0.05}")
# Confidence interval for the lift
from statsmodels.stats.proportion import proportion_confint
ci_a = proportion_confint(group_a["conversions"], group_a["visitors"], alpha=0.05)
ci_b = proportion_confint(group_b["conversions"], group_b["visitors"], alpha=0.05)
print(f"\n95% CI for A: {ci_a[0]:.1%} – {ci_a[1]:.1%}")
print(f"95% CI for B: {ci_b[0]:.1%} – {ci_b[1]:.1%}")
Common A/B Testing Mistakes¶
Peeking — stopping early when you see significance
Checking the test every day and stopping when p < 0.05 inflates your false positive rate. Run the test for the full planned duration.
Multiple testing
Running the same test on 10 segments and reporting only the ones that are significant gives you false positives. Apply Bonferroni correction: α/number_of_tests.
Not checking for novelty effects
A new feature might perform well initially because users are curious — not because it's genuinely better. Run tests for at least 2-4 weeks.
Selecting a biased sample
If you run an A/B test only on mobile users but conclude "this applies to all users," you may be wrong. Ensure your test sample represents the population.
Practice Exercises¶
Warm-up 1. Run a one-sample t-test: is the mean order value significantly different from £200? 2. Run a two-sample t-test: do Electronics orders have a significantly higher value than Books orders? 3. Calculate a 95% confidence interval for the mean order value.
Main 4. Simulate an A/B test: Group A has 2000 visitors and 240 conversions (12%). Group B has 2000 visitors and 290 conversions (14.5%). Is the difference statistically significant? 5. Calculate the 95% CI for the conversion rate in both groups. Do the intervals overlap? 6. Calculate the power of your A/B test: if the true effect is 2.5%, how likely were you to detect it?
Stretch 7. Your company has run the same A/B test for 5 days and is peeking at results daily. Simulate 1000 A/A tests (no real effect) to show how peeking inflates the false positive rate. 8. Design an A/B test for a new email subject line. Specify: sample size (use power analysis), metric, H₀ and H₁, significance level, duration.
Interview Questions¶
[Beginner] What is a p-value? What does p = 0.03 tell you?
[Mid-level] An A/B test shows p = 0.04. The product manager says "We can roll it out — it's statistically significant." What questions should you ask before agreeing?
[Senior] Explain the difference between statistical significance and practical significance. Give a business example where a result can be statistically significant but practically meaningless.