Generators & Memory Efficiency

Generators — Lazy Evaluation

A generator is like a list, but it generates values on-the-fly instead of storing them all in memory. Perfect for large datasets.

Generator Expressions

Use parentheses instead of brackets:

# List (stores all 1M items in memory)
squares_list = [x**2 for x in range(1_000_000)]

# Generator (generates each value when needed)
squares_gen = (x**2 for x in range(1_000_000))

Generator Functions

Use yield to create a generator function:

def count_up(n):
    i = 0
    while i < n:
        yield i      # "pause" and return value
        i += 1

for num in count_up(5):
    print(num)  # prints 0, 1, 2, 3, 4

Why Generators Matter

• Memory efficient: Process large files line-by-line instead of loading all into memory
• Lazy evaluation: Only compute what you need
• Infinite sequences: Can create endless generators

# Read a 10GB file without loading it all at once
with open("huge_dataset.csv") as f:
    for line in f:  # Implicit generator!
        process(line)

Data pipeline rule: If you're processing data that doesn't fit in memory, use generators.