Correlation Analysis

Correlation measures how strongly two variables move together. It's one of the most powerful (and most misused) tools in analytics. Used correctly, it helps you find the variables that matter. Used incorrectly, it leads to false conclusions.

Learning Objectives

• Calculate Pearson and Spearman correlation coefficients
• Interpret correlation strength and direction
• Build and read a correlation matrix
• Visualise correlations with heatmaps and scatter matrices
• Avoid the correlation/causation trap

---

Pearson Correlation Coefficient

Pearson's r measures linear correlation between two numeric variables: - +1: perfect positive linear relationship - 0: no linear relationship - -1: perfect negative linear relationship

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

df = pd.read_csv("orders_clean.csv")
df["total"] = df["quantity"] * df["unit_price"]

# Correlation between two specific variables
r = df["unit_price"].corr(df["total"])
print(f"Pearson correlation (unit_price vs total): r = {r:.3f}")

# P-value (is it statistically significant?)
from scipy import stats
r_value, p_value = stats.pearsonr(df["unit_price"].dropna(), df["total"].dropna())
print(f"r = {r_value:.3f}, p = {p_value:.4f}")
print(f"Significant: {p_value < 0.05}")

Interpreting correlation strength

| |r| | Interpretation | |-------|----------------| | 0.00–0.19 | Very weak or no relationship | | 0.20–0.39 | Weak relationship | | 0.40–0.59 | Moderate relationship | | 0.60–0.79 | Strong relationship | | 0.80–1.00 | Very strong relationship |

---

Correlation Matrix

numeric_df = df[["quantity", "unit_price", "total", "discount_pct"]].dropna()

# Compute correlation matrix
corr_matrix = numeric_df.corr()
print(corr_matrix.round(3))

Sample output:

	quantity	unit_price	total	discount_pct
quantity	1.000	-0.145	0.523	0.089
unit_price	-0.145	1.000	0.812	0.032
total	0.523	0.812	1.000	0.056
discount_pct	0.089	0.032	0.056	1.000

---

Correlation Heatmap

fig, ax = plt.subplots(figsize=(8, 6))
sns.heatmap(
    corr_matrix,
    annot=True,
    fmt=".2f",
    cmap="RdYlGn",
    center=0,
    vmin=-1, vmax=1,
    square=True,
    linewidths=0.5,
    ax=ax,
)
ax.set_title("Correlation Matrix")
plt.tight_layout()
plt.show()

---

Spearman Correlation — For Non-Linear Relationships

Pearson assumes a linear relationship. Spearman is rank-based — works for monotonic (but not necessarily linear) relationships and is robust to outliers.

# Spearman correlation
spearman_r = df["unit_price"].corr(df["total"], method="spearman")
print(f"Spearman correlation: {spearman_r:.3f}")

# Full correlation matrix with Spearman
spearman_matrix = numeric_df.corr(method="spearman")
print(spearman_matrix.round(3))

For mid-level analysts

Use Pearson when your data is approximately normally distributed and the relationship is linear. Use Spearman when: data is skewed, has outliers, or the relationship might be non-linear (just monotonic). For business analytics, Spearman is usually safer.

---

Correlation Between Numeric and Categorical

Use a box plot or ANOVA to understand whether group membership correlates with a numeric variable:

# Do different categories have different average totals?
from scipy.stats import f_oneway

groups = [group["total"].dropna().values for _, group in df.groupby("category")]
f_stat, p_val = f_oneway(*groups)
print(f"ANOVA: F = {f_stat:.3f}, p = {p_val:.4f}")
print(f"Category affects order total: {p_val < 0.05}")

---

The Correlation/Causation Warning

Correlation does not imply causation

Finding a correlation is a starting point, not an answer. Before acting on a correlation:

1. Is there a plausible mechanism? Why would X cause Y?
2. Is there a confounder? A third variable Z causing both X and Y?
3. Is it reverse causation? Maybe Y causes X, not the other way around.
4. Is it spurious? Many correlations are coincidental — especially with small datasets.

Real business example: You find that customers who opened marketing emails have 3x higher revenue. Before concluding "emails cause revenue," consider: customers who already intended to buy were more likely to open an email. The purchase intent caused both the email open AND the revenue.

---

Practice Exercises

Warm-up 1. Calculate the Pearson correlation between unit_price and total. Interpret the result. 2. Build and print the correlation matrix for all numeric columns. 3. Create a correlation heatmap.

Main 4. Compare Pearson vs Spearman correlation for unit_price vs total. Why might they differ? 5. Which two columns have the strongest correlation? Which have the weakest? What does each tell you? 6. Run an ANOVA test to determine whether category significantly affects total.

Stretch 7. Calculate rolling 30-day correlation between revenue and a secondary metric (e.g., number of orders). Does the correlation vary over time? 8. Create a scatter matrix (pd.plotting.scatter_matrix) for all numeric columns, coloured by status. Identify one non-obvious relationship.

---

Interview Questions

[Beginner] What does a Pearson correlation of -0.8 mean?

[Mid-level] When would you use Spearman correlation instead of Pearson?

[Senior] You find that ice cream sales and drowning deaths are both strongly correlated with temperature. How does this illustrate the correlation/causation problem in business analytics?

---

← Trend Analysis · Next: EDA Case Study →