Skip to content

Statistics — Interview Questions

Questions covering central tendency, spread, distributions, hypothesis testing, and A/B testing. These appear in data analyst and product analytics interviews at all levels.

[Beginner] What is the difference between mean, median, and mode? When would you use each?

Show answer
  • Mean: arithmetic average — affected by outliers. Use when distribution is symmetric.
  • Median: middle value — robust to outliers. Use when distribution is skewed (e.g., revenue, income).
  • Mode: most common value. Use for categorical data or when the most frequent value is meaningful.

Rule of thumb: if mean and median are far apart (ratio > 1.3), the distribution is skewed and median is more representative. Report both with an explanation.

print(f"Mean: {data.mean():.2f}")
print(f"Median: {data.median():.2f}")
if data.mean() / data.median() > 1.3:
    print("Right-skewed — prefer median for 'typical' value")

[Beginner] What is a p-value? What does p = 0.04 mean?

Show answer

A p-value is the probability of observing a result as extreme as (or more extreme than) the one you observed, assuming the null hypothesis is true.

p = 0.04 means: if there were truly no effect (H₀), you'd only see a result this extreme 4% of the time by random chance.

At a significance level of α = 0.05, p = 0.04 < 0.05, so you reject H₀ — there is statistically significant evidence of an effect.

Common mistake: p = 0.04 does NOT mean "there's a 96% probability the effect is real." That would be the posterior probability and requires Bayesian reasoning.

[Mid-level] What is the Central Limit Theorem and why does it matter for hypothesis testing?

Show answer

The CLT states that the distribution of sample means approaches a normal distribution as sample size increases — regardless of the shape of the underlying population distribution.

Why it matters: - Most hypothesis tests (t-test, z-test) assume the sampling distribution of the mean is normal - The CLT justifies this assumption for large samples (n > 30) even when raw data is skewed - This is why t-tests work on revenue data even though revenue is right-skewed — we're testing means, not individual values

# Demonstrate: sample means from a skewed distribution
skewed_data = np.random.exponential(1, 100_000)  # right-skewed
sample_means = [np.random.choice(skewed_data, 50).mean() for _ in range(2000)]
# sample_means follows a normal distribution (CLT in action)

[Mid-level] An A/B test on a new checkout button shows a 3% absolute improvement in conversion rate with p = 0.03. The product manager wants to roll out immediately. What questions do you ask first?

Show answer

  1. Sample size and duration: How many users were in each group? How long did the test run? A 3% lift with n=50 per group is unreliable; n=5000 is much stronger.

  2. Confidence interval: What's the CI for the 3% lift? If it's 0.5%-5.5%, the uncertainty is too wide to make a confident decision.

  3. Practical significance: Is 3% absolute lift worth the engineering cost to ship? What's the revenue impact if we scale? (3% × 100,000 monthly visitors × average order value = ?)

  4. Novelty effect: Did we run the test long enough to rule out users being curious about the new button?

  5. Segment consistency: Did the lift hold across key segments (mobile vs desktop, new vs returning users)? If only one segment improved and others got worse, the overall number is misleading.

  6. Guardrail metrics: Did any other metrics worsen? (e.g., revenue per conversion, return rate)

  7. Was the test properly randomised? No interference between groups (e.g., users in both groups)?

[Mid-level] What is the difference between statistical significance and practical significance?

Show answer

Statistical significance (low p-value): the observed result is unlikely to be due to random chance.

Practical significance: the effect is large enough to matter for the business.

A result can be: - Statistically significant but practically meaningless: with 1 million users, a 0.01% improvement in conversion is statistically detectable but generates £50/year — not worth building. - Practically significant but not statistically significant: a 10% improvement in a small study (n=30) might not reach p < 0.05 but could be worth running a larger test.

Always report effect size alongside the p-value: 

lift = rate_b - rate_a
ci_lower, ci_upper = ...  # confidence interval

print(f"Absolute lift: {lift:.1%} (95% CI: {ci_lower:.1%} to {ci_upper:.1%})")
print(f"Estimated annual revenue impact: £{lift * annual_visitors * avg_order:.0f}")
print(f"p-value: {p:.4f}")

[Senior] You're designing an A/B test to detect a 2% absolute improvement in conversion rate (baseline 12%). Walk me through how you'd determine the required sample size.

Show answer 
from statsmodels.stats.power import NormalIndPower
import numpy as np

baseline = 0.12     # current conversion rate
effect = 0.02       # minimum detectable effect (absolute)
alpha = 0.05        # significance level (5% false positive rate)
power = 0.80        # 80% chance of detecting a real effect

# Effect size (Cohen's h or standard proportion test)
p1 = baseline
p2 = baseline + effect
pooled_p = (p1 + p2) / 2
effect_size = abs(p2 - p1) / np.sqrt(pooled_p * (1 - pooled_p))

analysis = NormalIndPower()
n = analysis.solve_power(effect_size=effect_size, alpha=alpha, power=power)

print(f"Required per group: {int(np.ceil(n)):,}")
print(f"Total users needed: {int(np.ceil(n)) * 2:,}")

# If daily traffic is 500 users and 50% go to each group:
daily_per_group = 250
days_needed = int(np.ceil(n)) / daily_per_group
print(f"Test duration: {days_needed:.0f} days")

Additional considerations: - Run for a minimum of 1-2 full weeks to capture day-of-week effects - Don't peek at results and stop early (inflates false positives) - Pre-register the hypothesis and analysis plan before running - Ensure proper randomisation (users, not sessions, if possible)

[Senior] You run 20 A/B tests per month and 15% of them come back as significant (p < 0.05). Should you be confident in these results?

Show answer

No — this is the multiple testing problem. If you run 20 tests with α = 0.05, you'd expect 1 false positive by chance alone (20 × 0.05 = 1), even if none of the treatments have a real effect.

15% significant out of 20 tests = 3 significant results. But 1 of those is probably a false positive just from the number of tests run.

Solutions:

  1. Bonferroni correction: divide α by number of tests. For 20 tests, use α = 0.05/20 = 0.0025 as the significance threshold.

  2. Benjamini-Hochberg (FDR control): less conservative than Bonferroni, controls the expected proportion of false positives.

  3. Pre-register hypotheses: only test things you had a strong prior reason to believe would work — this reduces the number of tests and their false positive rate.

  4. Bayesian A/B testing: use a prior distribution on the effect size rather than null hypothesis significance testing — naturally accounts for multiple comparisons.

  5. Replication: significant results should be replicated in a follow-up test before full rollout.

← Practice Questions · Next: Week 02 — Power BI Basics →