Python for Data Analysis — Putting It Together

You now have Python fundamentals: variables, functions, lists, dicts, and file I/O. This file shows how these combine into real analytical work — before Pandas, and as a foundation for understanding what Pandas does under the hood.

Learning Objectives

• Write a complete data analysis pipeline in pure Python
• Calculate descriptive statistics from scratch
• Implement grouping, filtering, and aggregation without libraries
• Understand when to graduate from pure Python to Pandas
• Use Python's statistics and collections standard library modules

---

The Standard Library for Analytics

Python ships with useful modules — no pip install required.

import statistics      # mean, median, stdev, variance
import collections     # Counter, defaultdict
import itertools       # chain, groupby, combinations
import datetime        # date arithmetic
import math            # floor, ceil, sqrt, log

statistics module

import statistics

revenues = [42000, 38500, 51200, 49800, 55100, 48250, 61000, 44500]

print(statistics.mean(revenues))        # 48793.75
print(statistics.median(revenues))      # 49025.0
print(statistics.stdev(revenues))       # 7076.68
print(statistics.variance(revenues))    # 50080000.21
print(statistics.mode([1, 2, 2, 3, 3, 3]))  # 3

collections.Counter — frequency counting

from collections import Counter

statuses = ["completed", "pending", "completed", "cancelled",
            "completed", "pending", "completed", "refunded"]

counts = Counter(statuses)
print(counts)
# Counter({'completed': 4, 'pending': 2, 'cancelled': 1, 'refunded': 1})

print(counts.most_common(3))
# [('completed', 4), ('pending', 2), ('cancelled', 1)]

# Frequency as percentage
total = sum(counts.values())
for status, count in counts.most_common():
    print(f"{status}: {count} ({count/total:.1%})")

collections.defaultdict — group-by pattern

from collections import defaultdict

orders = [
    {"customer": "C001", "total": 298.00, "status": "completed"},
    {"customer": "C001", "total": 35.99,  "status": "completed"},
    {"customer": "C002", "total": 149.97, "status": "pending"},
    {"customer": "C003", "total": 389.00, "status": "completed"},
    {"customer": "C002", "total": 89.50,  "status": "completed"},
]

# Group by customer
by_customer = defaultdict(list)
for order in orders:
    by_customer[order["customer"]].append(order)

# Summarise each customer
for customer, customer_orders in sorted(by_customer.items()):
    completed = [o for o in customer_orders if o["status"] == "completed"]
    total_spend = sum(o["total"] for o in completed)
    print(f"{customer}: {len(completed)} completed orders, £{total_spend:,.2f} spent")

Output:

C001: 2 completed orders, £333.99 spent
C002: 1 completed orders, £89.50 spent
C003: 1 completed orders, £389.00 spent

---

Building an Analytics Pipeline

The dataset

import csv
from pathlib import Path
from collections import defaultdict
import statistics

# Sample dataset (normally loaded from CSV)
orders = [
    {"order_id": "O001", "customer_id": "C001", "product_id": "P001",
     "month": "2024-01", "quantity": 2, "unit_price": 149.00, "status": "completed"},
    {"order_id": "O002", "customer_id": "C003", "product_id": "P004",
     "month": "2024-01", "quantity": 1, "unit_price": 389.00, "status": "completed"},
    {"order_id": "O003", "customer_id": "C002", "product_id": "P003",
     "month": "2024-01", "quantity": 3, "unit_price": 49.99, "status": "pending"},
    {"order_id": "O004", "customer_id": "C001", "product_id": "P002",
     "month": "2024-02", "quantity": 1, "unit_price": 35.99, "status": "completed"},
    {"order_id": "O005", "customer_id": "C004", "product_id": "P001",
     "month": "2024-02", "quantity": 1, "unit_price": 149.00, "status": "cancelled"},
    {"order_id": "O006", "customer_id": "C003", "product_id": "P003",
     "month": "2024-02", "quantity": 5, "unit_price": 49.99, "status": "completed"},
]

# Add computed total
for o in orders:
    o["total"] = o["quantity"] * o["unit_price"]

Step 1 — Filter

completed = [o for o in orders if o["status"] == "completed"]
print(f"Completed orders: {len(completed)} of {len(orders)}")

Step 2 — Aggregate

totals = [o["total"] for o in completed]

print(f"Total revenue:  £{sum(totals):,.2f}")
print(f"Average order:  £{statistics.mean(totals):,.2f}")
print(f"Median order:   £{statistics.median(totals):,.2f}")
print(f"Largest order:  £{max(totals):,.2f}")
print(f"Smallest order: £{min(totals):,.2f}")

Step 3 — Group and compare

# Revenue by month
by_month = defaultdict(float)
for o in completed:
    by_month[o["month"]] += o["total"]

print("\nMonthly Revenue:")
for month in sorted(by_month):
    print(f"  {month}: £{by_month[month]:,.2f}")

# Revenue by product
by_product = defaultdict(lambda: {"orders": 0, "units": 0, "revenue": 0.0})
for o in completed:
    pid = o["product_id"]
    by_product[pid]["orders"] += 1
    by_product[pid]["units"] += o["quantity"]
    by_product[pid]["revenue"] += o["total"]

print("\nProduct Performance:")
for pid, data in sorted(by_product.items(), key=lambda x: -x[1]["revenue"]):
    print(f"  {pid}: {data['orders']} orders, {data['units']} units, £{data['revenue']:,.2f}")

Step 4 — Export results

import csv

summary_rows = [
    {"product_id": pid, **data}
    for pid, data in sorted(by_product.items(), key=lambda x: -x[1]["revenue"])
]

with open("product_summary.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["product_id", "orders", "units", "revenue"])
    writer.writeheader()
    writer.writerows(summary_rows)

---

Descriptive Statistics from Scratch

Understanding how these work helps you know when Pandas gives you the wrong answer.

def describe(data, label=""):
    """Compute descriptive statistics for a list of numbers."""
    n = len(data)
    if n == 0:
        return

    sorted_data = sorted(data)
    mean = sum(data) / n

    # Median
    mid = n // 2
    if n % 2 == 0:
        median = (sorted_data[mid - 1] + sorted_data[mid]) / 2
    else:
        median = sorted_data[mid]

    # Standard deviation (population)
    variance = sum((x - mean) ** 2 for x in data) / n
    std = variance ** 0.5

    # Percentiles
    def percentile(sorted_list, p):
        idx = (len(sorted_list) - 1) * p / 100
        lower = int(idx)
        frac = idx - lower
        if lower + 1 < len(sorted_list):
            return sorted_list[lower] + frac * (sorted_list[lower + 1] - sorted_list[lower])
        return sorted_list[lower]

    if label:
        print(f"\n{label}")
        print("-" * 30)
    print(f"Count:  {n}")
    print(f"Mean:   {mean:,.2f}")
    print(f"Median: {median:,.2f}")
    print(f"Std:    {std:,.2f}")
    print(f"Min:    {sorted_data[0]:,.2f}")
    print(f"25%:    {percentile(sorted_data, 25):,.2f}")
    print(f"75%:    {percentile(sorted_data, 75):,.2f}")
    print(f"Max:    {sorted_data[-1]:,.2f}")

describe([o["total"] for o in completed], "Order Totals")

---

When to Graduate from Pure Python to Pandas

Task	Pure Python	Pandas
Load a small CSV (< 10K rows)	Fine	Overkill
Load a large CSV (> 100K rows)	Slow	Use it
Filter rows	List comprehension	df[df["col"] > x]
Group and aggregate	defaultdict	df.groupby()
Merge two datasets	Manual loop	pd.merge()
Reshape data (pivot)	Very tedious	df.pivot_table()
Time series analysis	Difficult	Built-in
Plot a chart	Possible but verbose	df.plot()

Rule of thumb: if you have tabular data with more than a few thousand rows or more than 5-6 columns, switch to Pandas. The syntax overhead is worth it.

---

Practice Exercises

Warm-up 1. Using the orders list above, calculate the mean, median, and std of total for completed orders. Use the statistics module. 2. Use Counter to count how many orders fall into each status. 3. Use defaultdict to group orders by product_id and print the total revenue per product.

Main 4. Write a function monthly_summary(orders) that returns a dict of {month: {"revenue": ..., "orders": ..., "avg_order": ...}} for completed orders. 5. Write a function top_n_customers(orders, n=5) that returns the top N customers by total spend on completed orders, as a list of dicts sorted by spend descending. 6. Add a percentile_rank field to each completed order: what percentage of completed orders have a smaller total than this one?

Stretch 7. Implement a simple two-pass algorithm that detects outlier orders: any order whose total is more than 2 standard deviations above the mean is flagged as an outlier. 8. Write a complete mini-pipeline: load orders from a CSV, filter to completed, group by customer, compute summary stats per customer, and export to a new CSV — all in pure Python with no Pandas.

---

← File Handling · Next: Practice Exercises →