Variance and Standard Deviation¶
The mean tells you where the centre is. Variance and standard deviation tell you how spread out the data is around that centre. Two datasets can have the same mean but completely different variability — and that variability often matters more for business decisions.
Learning Objectives¶
- Compute and interpret variance and standard deviation
- Understand the difference between population and sample statistics
- Use the coefficient of variation to compare spread across datasets
- Apply standard deviation in business contexts
Variance — Average Squared Deviation from the Mean¶
import numpy as np
# Daily revenue for two different months
jan = [42000, 43000, 41500, 44000, 42500, 43500, 41000]
mar = [20000, 65000, 35000, 72000, 18000, 48000, 62000]
print(f"Jan mean: £{np.mean(jan):,.0f} | Mar mean: £{np.mean(mar):,.0f}")
print(f"Jan variance: £{np.var(jan, ddof=1):,.0f} | Mar variance: £{np.var(mar, ddof=1):,.0f}")
Both months average £42,500 — but March is wildly unpredictable compared to January.
Variance = average of squared distances from the mean:
The squared distances mean: (1) positive and negative deviations don't cancel, (2) large deviations get extra weight.
Problem: variance is in squared units (£²). Standard deviation fixes this.
Standard Deviation — Spread in the Original Units¶
jan_std = np.std(jan, ddof=1) # sample std (use ddof=1 for samples)
mar_std = np.std(mar, ddof=1)
print(f"Jan std: £{jan_std:,.0f}") # ≈ £1,041
print(f"Mar std: £{mar_std:,.0f}") # ≈ £21,154
Now we can say: "March daily revenue varies by about £21,154 around the mean, while January varies by only £1,041." That's actionable.
Population vs Sample standard deviation¶
# Population std (ddof=0): when you have ALL the data
pop_std = np.std(data, ddof=0)
# Sample std (ddof=1): when your data is a sample from a larger population
sample_std = np.std(data, ddof=1)
In analytics, you almost always have a sample (not the entire population of all possible values). Use ddof=1 (Bessel's correction) for sample data.
Interpreting Standard Deviation¶
For a normally distributed variable: - ~68% of values fall within 1 std of the mean - ~95% fall within 2 std - ~99.7% fall within 3 std
mean_rev = np.mean(jan)
std_rev = np.std(jan, ddof=1)
print(f"68% of days: £{mean_rev - std_rev:,.0f} to £{mean_rev + std_rev:,.0f}")
print(f"95% of days: £{mean_rev - 2*std_rev:,.0f} to £{mean_rev + 2*std_rev:,.0f}")
The 68-95-99.7 rule only applies to normal distributions
Revenue and prices are typically right-skewed. For these, use percentiles instead: np.percentile(data, [5, 25, 75, 95]).
Coefficient of Variation — Comparing Spread Across Datasets¶
Std can't be compared between datasets with different means. A £1,000 std on a £100,000 metric is tiny; on a £2,000 metric it's huge.
Coefficient of Variation (CV) = std / mean — normalises for scale.
jan_cv = np.std(jan, ddof=1) / np.mean(jan)
mar_cv = np.std(mar, ddof=1) / np.mean(mar)
print(f"Jan CV: {jan_cv:.2%}") # 2.5% — very stable
print(f"Mar CV: {mar_cv:.2%}") # 49.8% — highly variable
Lower CV = more consistent. Higher CV = more variable.
# Compare variability across product categories
category_cv = (
df.groupby("category")["total"]
.agg(mean="mean", std="std")
.assign(cv=lambda x: x["std"] / x["mean"])
.sort_values("cv", ascending=False)
)
print(category_cv)
Business Applications¶
Process control — is variability acceptable?¶
# Delivery time SLA: mean ≤ 3 days, std ≤ 0.5 days
delivery_times = [2.1, 3.2, 2.8, 3.5, 2.4, 3.1, 2.9, 4.2, 2.7, 3.0]
print(f"Mean: {np.mean(delivery_times):.1f} days")
print(f"Std: {np.std(delivery_times, ddof=1):.2f} days")
if np.std(delivery_times, ddof=1) > 0.5:
print("⚠ Delivery time variability exceeds SLA threshold")
Anomaly detection — z-score¶
# Z-score: how many standard deviations from the mean?
daily_orders = pd.read_csv("daily_orders.csv")["orders"]
z_scores = (daily_orders - daily_orders.mean()) / daily_orders.std()
anomalies = daily_orders[np.abs(z_scores) > 2]
print(f"Anomalous days: {len(anomalies)}")
Practice Exercises¶
Warm-up
1. Calculate mean, variance, and std for [89, 125, 149, 165, 189, 210, 245, 310].
2. Calculate CV for order totals in two different product categories.
3. For which of these two datasets is the mean a more reliable predictor of the next value? A = [100, 101, 99, 102, 100] vs B = [50, 150, 80, 200, 20].
Main 4. Calculate std for each product category's order value. Which has the most consistent pricing? 5. Find days where daily revenue is more than 2 standard deviations from the mean (anomaly detection). 6. Create a figure showing two histograms with the same mean but different standard deviations.
Stretch
7. Build a "volatility dashboard": for each product, show mean order value, std, CV, and a classification of Low/Medium/High volatility.
8. Explain why ddof=1 (Bessel's correction) gives a better estimate of population variance from a sample. When would you use ddof=0?
Interview Questions¶
[Beginner] What is standard deviation? Explain it without using the word "average."
[Mid-level] What is the coefficient of variation and when is it more useful than standard deviation?
[Senior] Two investment portfolios have the same expected return (mean) but different standard deviations. How would you use this information to advise a client?