Outliers

An outlier is a value that lies far outside the typical range of the data. Outliers can be legitimate extreme values (a corporate client who spends 100x the average), data entry errors (a price entered as 14900 instead of 149.00), or system glitches (a sensor returning −9999 as a null indicator). The first step is always to understand which type you are dealing with — before touching anything.

Why outliers matter

A single £500,000 order (data entry error: quantity of 5,000 instead of 5) will inflate average order value from £250 to £750. Every metric derived from that average — sales rep targets, category benchmarks, forecasting models — will be wrong. Outlier detection is not optional.

Learning Objectives

• Detect outliers using IQR, z-score, and visualisation
• Distinguish between legitimate extreme values and data entry errors
• Apply appropriate treatment: keep, cap, transform, or remove
• Understand the impact of outliers on statistical analysis

---

Detecting Outliers — IQR Method

The Interquartile Range (IQR) method is robust to extreme values and works well for skewed distributions — which most business data follows.

import pandas as pd
import numpy as np

df = pd.read_csv("sales_data.csv")
df["total_amount"] = df["quantity"] * df["unit_price"]

# Calculate IQR for total_amount
Q1 = df["total_amount"].quantile(0.25)
Q3 = df["total_amount"].quantile(0.75)
IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR

print(f"Q1:          £{Q1:,.2f}")
print(f"Q3:          £{Q3:,.2f}")
print(f"IQR:         £{IQR:,.2f}")
print(f"Lower bound: £{lower_bound:,.2f}")
print(f"Upper bound: £{upper_bound:,.2f}")

# Flag outliers
outlier_mask = (df["total_amount"] < lower_bound) | (df["total_amount"] > upper_bound)
outliers = df[outlier_mask]

print(f"\nOutliers detected: {len(outliers):,} rows ({len(outliers)/len(df):.1%} of data)")
print(outliers[["order_id", "customer_id", "product_name", "quantity", "unit_price", "total_amount"]].head(10))

Expected output:

Q1:          £89.97
Q3:          £298.00
IQR:         £208.03
Lower bound: £-222.07 (negative — floor at 0)
Upper bound: £610.05

Outliers detected: 47 rows (4.7% of data)

For mid-level analysts

The 1.5 × IQR multiplier is a convention, not a law. For business data where extreme values are expected (wholesale orders, enterprise contracts), use 3 × IQR to catch only the most extreme cases. For tightly controlled data (sensor readings, transaction fees), use 1.0 × IQR for a stricter threshold.

# Reusable function for IQR outlier detection
def detect_outliers_iqr(series, multiplier=1.5):
    Q1 = series.quantile(0.25)
    Q3 = series.quantile(0.75)
    IQR = Q3 - Q1
    lower = Q1 - multiplier * IQR
    upper = Q3 + multiplier * IQR
    return series < lower, series > upper, lower, upper

low_mask, high_mask, lb, ub = detect_outliers_iqr(df["total_amount"])
print(f"Below lower bound: {low_mask.sum()} rows")
print(f"Above upper bound: {high_mask.sum()} rows")

---

Detecting Outliers — Z-Score Method

Z-score measures how many standard deviations a value is from the mean. Values with |z| > 3 are typically flagged.

from scipy import stats

# Z-score — assumes approximately normal distribution
z_scores = np.abs(stats.zscore(df["total_amount"].dropna()))
outlier_mask_z = z_scores > 3

print(f"Z-score outliers (|z| > 3): {outlier_mask_z.sum():,}")

Z-score assumes a normal distribution

Revenue and order values are almost always right-skewed — the z-score method will flag far fewer upper outliers than IQR because the mean and standard deviation are themselves inflated by the outliers. For skewed business data, use IQR. Z-score is appropriate for approximately normally distributed measurements (test scores, heights, sensor error rates).

# Comparing IQR vs z-score: how many outliers does each method find?
iqr_outliers = df[outlier_mask].shape[0]
z_outliers = outlier_mask_z.sum()

print(f"IQR method (1.5x): {iqr_outliers} outliers ({iqr_outliers/len(df):.1%})")
print(f"Z-score method (|z|>3): {z_outliers} outliers ({z_outliers/len(df):.1%})")

---

Visualising Outliers

Looking at outliers visually reveals their structure — a boxplot shows you there are outliers, a histogram shows you where they cluster.

import matplotlib.pyplot as plt
import seaborn as sns

fig, axes = plt.subplots(1, 3, figsize=(16, 5))

# Boxplot — shows quartiles and outlier points (dots beyond whiskers)
axes[0].boxplot(df["total_amount"].dropna(), vert=True, patch_artist=True,
                boxprops=dict(facecolor="#0D9488", alpha=0.7))
axes[0].set_title("Order Total — Boxplot")
axes[0].set_ylabel("£")

# Histogram with IQR boundaries
axes[1].hist(df["total_amount"].dropna(), bins=50, edgecolor="white", color="#0D9488", alpha=0.8)
axes[1].axvline(ub, color="red", linestyle="--", label=f"Upper IQR bound £{ub:,.0f}")
axes[1].axvline(df["total_amount"].mean(), color="orange", linestyle="--", label="Mean")
axes[1].set_title("Order Total — Distribution")
axes[1].legend(fontsize=8)

# Log-transformed histogram (compresses the tail — reveals the full distribution)
axes[2].hist(np.log1p(df["total_amount"].dropna()), bins=50, edgecolor="white", color="#F59E0B", alpha=0.8)
axes[2].set_title("Order Total — Log Scale")
axes[2].set_xlabel("log(1 + total_amount)")

plt.suptitle("Outlier Detection — Order Total", fontsize=13)
plt.tight_layout()
plt.show()

---

Investigate Before Acting

Never remove or modify an outlier without first understanding it. Print the outlier rows and read them.

# Examine the top 10 largest orders
top_orders = df.nlargest(10, "total_amount")[
    ["order_id", "customer_id", "product_name", "category", "quantity", "unit_price", "total_amount"]
]
print(top_orders.to_string())

Ask these questions about each outlier:

Question	Answer → Action
Is the quantity suspiciously large? (e.g., 1,000 units of a luxury item)	Likely data entry error — investigate
Is the unit_price 10x the typical range?	Likely data entry error — compare to other rows for same product
Is this a known corporate client?	Legitimate bulk order — keep and flag
Does this order appear in the source system?	Cross-reference with ERP/CRM to verify
Is this the only order from this customer?	No history = more likely to be an error

Investigate before removing

Never delete outliers without examining them. A £50,000 order might be a data entry error — or it might be your biggest corporate client. Look at the full row, check the source system if possible, and document your decision either way.

---

Treatment Strategies

Keep — when the outlier is a legitimate extreme value

# Flag for downstream users — do not remove
df["is_large_order"] = (df["total_amount"] > ub).astype(int)
print(f"Large orders flagged: {df['is_large_order'].sum()}")

# These orders are real — just extreme. Your dashboard can offer a filter for them.

Cap (Winsorize) — clip to a boundary while preserving the row

# Cap at the IQR upper bound — preserves row count, reduces extreme skew
df["total_capped"] = df["total_amount"].clip(lower=max(lb, 0), upper=ub)

# Or cap at a specific business-rule threshold
df["total_capped"] = df["total_amount"].clip(upper=5000)

# Compare mean before and after capping
print(f"Mean before capping: £{df['total_amount'].mean():,.2f}")
print(f"Mean after capping:  £{df['total_capped'].mean():,.2f}")

For mid-level analysts

Winsorising is common in financial data analysis. It preserves the sample size (unlike dropping) while preventing extreme values from distorting aggregates. The 95th/99th percentile is often used as the cap, rather than the IQR bound, when you want to preserve more of the natural range.

# Winsorise at 1st and 99th percentiles
p01 = df["total_amount"].quantile(0.01)
p99 = df["total_amount"].quantile(0.99)
df["total_winsorised"] = df["total_amount"].clip(lower=p01, upper=p99)
print(f"Winsorised at £{p01:.2f} – £{p99:.2f}")

Remove — when the value is clearly a data entry error

# Remove rows where unit_price is impossible for the product category
# e.g., a book priced at £14,900 is almost certainly £149.00 entered as £14,900
before = len(df)
df_clean = df[df["unit_price"] <= 10_000]    # business rule: no consumer product > £10k
print(f"Removed {before - len(df_clean):,} rows with impossible unit prices")

Transform — log-transform to reduce the influence of extreme values

# Log transformation is for analysis/modelling — keep original values in your main dataset
df["log_total"] = np.log1p(df["total_amount"])  # log1p = log(1 + x), handles 0s correctly

# Compare distributions
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 4))

df["total_amount"].hist(bins=40, ax=ax1, color="#0D9488", edgecolor="white")
ax1.set_title("Original Distribution (right-skewed)")
ax1.set_xlabel("£")

df["log_total"].hist(bins=40, ax=ax2, color="#F59E0B", edgecolor="white")
ax2.set_title("Log-Transformed (more symmetric)")
ax2.set_xlabel("log(1 + £)")

plt.tight_layout()
plt.show()

For senior analysts

Log transformation is widely used in regression models for right-skewed targets like revenue, salary, and price. When you log-transform a variable, the model learns multiplicative rather than additive effects — a 10% change in X drives a proportional change in Y. When interpreting coefficients from a log-transformed model, remember to exponentiate back.

---

Multivariate Outliers

Sometimes a value looks normal individually but is anomalous in combination:

# A quantity of 500 is fine for stationery but suspicious for laptops (£500 each)
electronics = df[df["category"] == "Electronics"]
suspicious_bulk = electronics[electronics["quantity"] > 50]

print(f"Bulk electronics orders (quantity > 50): {len(suspicious_bulk)}")
print(suspicious_bulk[["order_id", "product_name", "quantity", "unit_price", "total_amount"]].to_string())

# Flag multivariate outliers
median_qty = df.groupby("category")["quantity"].transform("median")
median_price = df.groupby("category")["unit_price"].transform("median")

df["suspicious"] = (df["quantity"] > median_qty * 5) & (df["unit_price"] > median_price * 3)
print(f"\nMultivariate suspicious orders: {df['suspicious'].sum()}")

---

Documenting Your Outlier Decisions

# Keep a record of what you did and why
outlier_log = []

def log_outlier_decision(order_ids, action, reason):
    for oid in order_ids:
        outlier_log.append({
            "order_id": oid,
            "action": action,
            "reason": reason,
        })

# Example decisions
log_outlier_decision(
    ["O0412"],
    action="REMOVED",
    reason="unit_price = £14,900 for a book. Confirmed data entry error (extra zero). Correct price is £149.00 per product catalog.",
)
log_outlier_decision(
    ["O0871", "O0943"],
    action="FLAGGED as is_large_order",
    reason="Legitimate bulk orders from corporate client ACC-Corp. Verified in CRM. Keep for revenue analysis, exclude from AOV benchmarking.",
)

outlier_df = pd.DataFrame(outlier_log)
print(outlier_df.to_string())
# Save to file for audit trail
outlier_df.to_csv("outlier_decisions.csv", index=False)

---

Practice Exercises

Warm-up

1. Calculate Q1, Q3, and IQR for the unit_price column. Print the lower and upper IQR bounds.
2. Count how many rows fall outside the IQR boundaries for total_amount.
3. Create a boxplot of unit_price to visualise the distribution and outliers.

Main

1. Flag all orders where total_amount > upper_bound with a new column is_large_order (1 or 0). How many large orders are there? What is their combined revenue?
2. Winsorise total_amount at the 1st and 99th percentiles. Create a new column total_winsorised. Compare mean and median before and after.
3. Log-transform total_amount with np.log1p(). Plot the original and transformed distributions side by side. Comment on how the shape changed.

Stretch

1. Write a function detect_outliers(df, col, method="iqr", multiplier=1.5) that:
2. Accepts "iqr" or "zscore" as the method
3. Returns a DataFrame of outlier rows with an added outlier_type column ("low" or "high")
4. Prints a summary: bounds used, number of outliers, percentage of total rows
5. Identify multivariate outliers: orders where quantity > 5 × category_median_quantity AND unit_price > 3 × category_median_price. Print these rows. For each, write a one-line comment on whether it is likely an error or a legitimate order.

---

Interview Questions

[Beginner] What is an outlier? How do you detect one in a dataset?

Show answer

An outlier is a data point that lies significantly outside the typical range of values in a column. In a column of order totals where 99% of values are between £10 and £500, an order of £50,000 is an outlier.

Detection methods:

Visual: - Boxplot: data points beyond the whiskers are flagged as outliers - Histogram: a long tail or isolated bars far from the main cluster

Statistical: - IQR method: flag values below Q1 - 1.5 * IQR or above Q3 + 1.5 * IQR - Z-score: flag values where |z| > 3 (3 standard deviations from the mean)

Q1 = df["total_amount"].quantile(0.25)
Q3 = df["total_amount"].quantile(0.75)
IQR = Q3 - Q1
outliers = df[(df["total_amount"] < Q1 - 1.5 * IQR) | (df["total_amount"] > Q3 + 1.5 * IQR)]
print(f"{len(outliers)} outliers detected")

---

[Beginner] Should you always remove outliers?

Show answer

No. Removing outliers without investigation is a mistake.

Outliers can be: - Legitimate extreme values — a whale customer who genuinely spends £50,000/month - Data entry errors — a quantity of 10,000 that should have been 100 - System artefacts — a sensor returning −9999 as a null placeholder

The correct process is: 1. Detect outliers (IQR or z-score) 2. Investigate each: look at the full row, check the source system 3. Decide: keep (and flag), cap, remove, or transform 4. Document the decision and the reason

Silently removing outliers changes the population you're analysing and can hide important business information — like the fact that 2% of customers generate 30% of revenue.

---

[Mid-level] What is the IQR method for outlier detection? Why is it more robust than z-score for skewed data?

Show answer

IQR method: 1. Calculate Q1 (25th percentile) and Q3 (75th percentile) 2. IQR = Q3 − Q1 (the range of the middle 50% of the data) 3. Lower bound = Q1 − 1.5 × IQR 4. Upper bound = Q3 + 1.5 × IQR 5. Flag values outside [lower bound, upper bound]

Why it is more robust: The IQR uses percentiles (rank-based), which are not affected by extreme values. If your data has 10 orders worth £500,000 (outliers), they do not shift Q1 or Q3 — the bounds stay accurate.

The z-score uses mean and standard deviation, which are both inflated by extreme values. A dataset with several £500,000 orders will have a high mean and high std — making the upper z-score threshold so large that even moderate outliers pass the test. The z-score "masks" outliers by inflating the threshold with the outliers themselves.

Use IQR for skewed distributions (revenue, prices, wait times). Use z-score only when the distribution is approximately normal.

---

[Mid-level] You find that 3% of orders have a total_amount > £5,000. How do you decide what to do with them?

Show answer

1. Examine the rows — print the full record including product_name, quantity, unit_price, customer_id, and status. Look for patterns:
2. Are they all the same customer? (possible bulk client)
3. Are they all the same product? (high-value product)

4. Do the quantity × unit_price calculations make sense?

5. Cross-reference — check the same orders in the source CRM or ERP. Are they real?

6. Analyse the distribution — plot these high-value orders separately. If they cluster around £5,000–£8,000, they may be a legitimate corporate segment. If there are a few at £50,000–£500,000, those are more suspicious.

7. Apply a business rule:

8. If confirmed real: keep, add is_large_order = 1 flag, exclude from Average Order Value benchmarks (or report separately)
9. If data entry errors: correct where possible (divide by 10 if the error is an extra zero), remove if not correctable

10. If uncertain: flag for manual review, exclude from automated reporting until resolved

11. Document — record every decision in a cleaning log with the reason.

---

[Senior] A colleague removes all orders in the 99th percentile before calculating the average order value. What is wrong with this approach, and what should they do instead?

Show answer

What is wrong: 1. It removes legitimate high-value customers — the top 1% of orders might include your most important clients. Removing them from AOV calculations creates a metric that does not represent actual business performance. 2. It is undocumented and irreproducible — if a different analyst calculates AOV without removing the 99th percentile, the numbers will not match. Analysts comparing metrics will be confused. 3. It conflates outlier detection with outlier removal — some of those 99th percentile orders may be data entry errors; others may be legitimate. Applying a blanket rule removes both. 4. The threshold is arbitrary — why 99th? Why not 95th or 99.5th?

What they should do instead: 1. Detect, investigate, then decide — examine the high-value orders and classify each as: legitimate, error, or uncertain. 2. Create separate metrics — report both: - "AOV including all orders: £250" - "AOV excluding corporate bulk orders (>£5,000): £195" 3. Flag, not remove — add an is_large_order column. Filter in dashboards when the question specifically requires the typical order. Keep all rows in the underlying dataset. 4. Document the definition — "AOV is defined as median order value for orders under £5,000, which represents 97% of transaction volume."

---

Previous: Duplicates | Next: Data Transformation