File Handling — Reading and Writing Data

Data analysis starts with getting data in and ends with getting results out. Python's file handling covers both — from reading a CSV you downloaded from a database export to writing a cleaned dataset for the next step in your pipeline.

Learning Objectives

• Read and write plain text files
• Read CSV files with the csv module
• Write CSV files
• Work with file paths using pathlib
• Handle common file errors gracefully
• Understand when to use the csv module vs Pandas (covered in Part 2)

---

Reading Plain Text Files

# Read entire file at once
with open("report.txt", "r", encoding="utf-8") as f:
    content = f.read()
    print(content)

# Read line by line (memory-efficient for large files)
with open("report.txt", "r", encoding="utf-8") as f:
    for line in f:
        print(line.strip())   # .strip() removes trailing newline

# Read all lines into a list
with open("report.txt", "r", encoding="utf-8") as f:
    lines = f.readlines()
print(f"Total lines: {len(lines)}")

Always use with open(...) — the context manager

The with block automatically closes the file when you're done, even if an error occurs. Never use f = open(...) without a corresponding f.close().

---

Writing Plain Text Files

report_lines = [
    "Monthly Revenue Report",
    "=" * 30,
    "January:  £42,000",
    "February: £38,500",
    "March:    £51,200",
]

# Write (overwrites if file exists)
with open("report.txt", "w", encoding="utf-8") as f:
    f.write("\n".join(report_lines))

# Append to existing file
with open("report.txt", "a", encoding="utf-8") as f:
    f.write("\nApril:    £49,800\n")

File modes: "r" (read), "w" (write, overwrites), "a" (append), "r+" (read + write)

---

Reading CSV Files — The csv Module

For simple CSVs without Pandas:

import csv

with open("sales_data.csv", "r", newline="", encoding="utf-8") as f:
    reader = csv.DictReader(f)   # reads header row as column names

    for row in reader:
        # Each row is a dict: {"order_id": "O001", "total": "298.00", ...}
        order_id = row["order_id"]
        total = float(row["total"])   # CSV values are always strings — convert!
        print(f"{order_id}: £{total:,.2f}")

All CSV values are strings — always convert

row["total"] returns "298.00" (a string), not 298.00 (a float). Use float(), int(), or datetime.strptime() to convert.

Reading into a list of dicts

import csv

def load_csv(filepath):
    """Load a CSV file and return a list of dicts."""
    with open(filepath, "r", newline="", encoding="utf-8") as f:
        return list(csv.DictReader(f))

orders = load_csv("sales_data.csv")
print(f"Loaded {len(orders)} orders")
print(orders[0])   # {'order_id': 'O001', 'customer_id': 'C001', ...}

---

Writing CSV Files

import csv

results = [
    {"customer_id": "C001", "total_orders": 9, "total_revenue": 1456.25},
    {"customer_id": "C003", "total_orders": 12, "total_revenue": 2195.50},
    {"customer_id": "C012", "total_orders": 8, "total_revenue": 2840.00},
]

with open("customer_summary.csv", "w", newline="", encoding="utf-8") as f:
    fieldnames = ["customer_id", "total_orders", "total_revenue"]
    writer = csv.DictWriter(f, fieldnames=fieldnames)

    writer.writeheader()     # write column names as first row
    writer.writerows(results)

print("Written to customer_summary.csv")

newline="" in the open() call

On Windows, not passing newline="" causes extra blank lines in the CSV. Always include it when writing CSV files.

---

Working with File Paths — pathlib

pathlib.Path is the modern, cross-platform way to work with file paths. It works on Windows (\), Mac and Linux (/) without any special handling.

from pathlib import Path

# Current directory
here = Path(".")
print(here.resolve())   # absolute path to current directory

# Navigate paths
data_dir = Path("data")
file = data_dir / "sales_data.csv"   # joins paths correctly on all OS
print(file)   # data/sales_data.csv (or data\sales_data.csv on Windows)

# Check if file exists
if file.exists():
    print(f"File found: {file}")
else:
    print("File not found")

# List all CSV files in a directory
csv_files = list(Path("data").glob("*.csv"))
print(f"Found {len(csv_files)} CSV files")

# Get file info
print(file.name)        # sales_data.csv
print(file.stem)        # sales_data
print(file.suffix)      # .csv
print(file.parent)      # data

Reading all CSVs in a folder

import csv
from pathlib import Path

all_orders = []
for csv_path in Path("monthly_exports").glob("*.csv"):
    with open(csv_path, "r", newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        month_orders = list(reader)
        print(f"{csv_path.stem}: {len(month_orders)} orders")
        all_orders.extend(month_orders)

print(f"Total orders loaded: {len(all_orders)}")

---

Error Handling

Files fail. The data directory doesn't exist. The file is locked. The encoding is wrong. Handle these gracefully.

from pathlib import Path

def safe_load_csv(filepath):
    """Load CSV file with error handling."""
    path = Path(filepath)

    if not path.exists():
        print(f"File not found: {filepath}")
        return []

    if path.stat().st_size == 0:
        print(f"File is empty: {filepath}")
        return []

    try:
        import csv
        with open(path, "r", newline="", encoding="utf-8") as f:
            return list(csv.DictReader(f))
    except UnicodeDecodeError:
        # Try with a different encoding (common with Windows-exported CSVs)
        with open(path, "r", newline="", encoding="cp1252") as f:
            return list(csv.DictReader(f))
    except csv.Error as e:
        print(f"CSV parsing error in {filepath}: {e}")
        return []

For mid-level analysts

Windows-exported CSVs (from Excel "Save As CSV") often use Windows-1252 encoding (cp1252), not UTF-8. This causes UnicodeDecodeError on special characters like £, €, or accented letters. Always handle both encodings.

---

A Realistic Pipeline — Load, Process, Export

import csv
from pathlib import Path
from collections import defaultdict

def load_orders(filepath):
    with open(filepath, "r", newline="", encoding="utf-8") as f:
        reader = csv.DictReader(f)
        return [
            {
                **row,
                "total": float(row["quantity"]) * float(row["unit_price"]),
            }
            for row in reader
            if row["status"] == "completed"
        ]

def summarise_by_customer(orders):
    summary = defaultdict(lambda: {"orders": 0, "revenue": 0.0})
    for order in orders:
        cid = order["customer_id"]
        summary[cid]["orders"] += 1
        summary[cid]["revenue"] += order["total"]
    return dict(summary)

def export_summary(summary, filepath):
    rows = [
        {"customer_id": cid, "orders": data["orders"], "revenue": round(data["revenue"], 2)}
        for cid, data in sorted(summary.items(), key=lambda x: -x[1]["revenue"])
    ]
    with open(filepath, "w", newline="", encoding="utf-8") as f:
        writer = csv.DictWriter(f, fieldnames=["customer_id", "orders", "revenue"])
        writer.writeheader()
        writer.writerows(rows)

# Run the pipeline
orders = load_orders("orders.csv")
summary = summarise_by_customer(orders)
export_summary(summary, "customer_revenue_summary.csv")
print(f"Processed {len(orders)} orders for {len(summary)} customers.")

---

Practice Exercises

Warm-up 1. Write code that reads a text file line by line and prints lines that start with "Error:". 2. Write a CSV file with 3 columns (name, score, grade) and 5 rows of made-up data. 3. Use pathlib.Path to check if a file called data.csv exists in the current directory.

Main 4. Write a function that loads a CSV of orders and returns only orders where total > 100 as a list of dicts. Ensure all numeric columns are properly typed. 5. Write a function that takes a list of order dicts and exports them to a CSV, sorted by total descending. 6. Load all CSV files from a directory, combine them into one list, and write the combined data to a single output CSV.

Stretch 7. Write a function that loads a CSV, detects whether it's UTF-8 or CP1252 encoded, and handles both without crashing. 8. Write a function that compares two CSV files (same columns) and outputs a third CSV containing only rows that are in file A but NOT in file B, based on a key column.

---

Interview Questions

[Beginner] Why should you use with open(...) instead of f = open(...)?

[Beginner] Why do you need to convert CSV values to int or float after reading them?

[Mid-level] What is pathlib.Path and why is it preferred over string path manipulation (os.path.join)?

[Mid-level] You load a CSV on Windows and get UnicodeDecodeError. What is the likely cause and how do you fix it?

[Senior] You need to process a 10GB CSV file without loading it all into memory. How would you do this in Python?

---

← Lists and Dictionaries · Next: Python for Data Analysis →