Advanced Iteration & Decoration

Closures, decorators, iterators, and generators are among Python’s most powerful advanced features. While they are highly effective on their own, combining them allows you to build elegant, memory-efficient, and easily monitorable applications. For example, you can use decorators to profile or rate-limit generator streams, or use closures to maintain state inside custom iteration logic.

Learning Objectives

After completing this lesson, you will be able to:

Apply decorators to generator functions to monitor, time, or log value generation.
Understand the execution flow differences when decorating normal functions vs. generator functions.
Combine multiple generators into data pipelines and manage them using decorators.
Use closures to configure custom iterators dynamically.

Pattern 1: Decorating Generators

Decorators are commonly used to log, time, or rate-limit functions. However, when you apply a decorator to a generator function, you must be careful about when code executes.

The Generator Decorator Nuance

Recall that calling a generator function does not execute its body; it immediately returns a generator object. If a standard decorator wraps a generator, the wrapper function will run when the generator is created, but not when values are yielded. To intercept the actual value generation, the decorator’s wrapper must iterate through the generator and yield the values.

Example: Yield Logger

Here is a decorator that logs every time a generator yields a value:

def log_yields(func):
    def wrapper(*args, **kwargs):
        generator = func(*args, **kwargs)
        print(f"[Decorator] Generator '{func.__name__}' initialized.")
        
        # We must iterate and yield to preserve lazy evaluation
        for index, value in enumerate(generator, start=1):
            print(f"[Decorator] Yielding item #{index}: {value}")
            yield value
            
    return wrapper

@log_yields
def count_to_three():
    yield "One"
    yield "Two"
    yield "Three"

# Test the decorated generator
gen = count_to_three()
print("--- Starting Iteration ---")
for item in gen:
    pass

Output

[Decorator] Generator 'count_to_three' initialized.
--- Starting Iteration ---
[Decorator] Yielding item #1: One
[Decorator] Yielding item #2: Two
[Decorator] Yielding item #3: Three

Practice

Exercise 1

Predict the output of the following code:

def print_call(func):
    def wrapper(*args, **kwargs):
        print("Decorator Wrapper Called")
        return func(*args, **kwargs)
    return wrapper

@print_call
def my_generator():
    print("Generator Body Started")
    yield 10

g = my_generator()
print("Generator Created")
print(next(g))

Solution

Decorator Wrapper Called
Generator Created
Generator Body Started
10

Calling my_generator() immediately triggers the wrapper print_call which prints "Decorator Wrapper Called" and returns the generator object. The code inside my_generator() (printing "Generator Body Started") runs only when next(g) is called.

Pattern 2: Stateful Iterators with Closures

Closures can be used to dynamically configure custom iterators without defining a full class structure. By combining a closure with a generator, we can maintain configuration and iteration state cleanly.

Example: Configurable Fibonacci Generator

def make_fibonacci_generator(max_value):
    # Enclosed configuration state
    limit = max_value
    
    def fib():
        a, b = 0, 1
        while a <= limit:
            yield a
            a, b = b, a + b
            
    return fib

# Create two independent generators with different limits
fib_up_to_10 = make_fibonacci_generator(10)()
fib_up_to_50 = make_fibonacci_generator(50)()

print("Fibonacci to 10:", list(fib_up_to_10))
print("Fibonacci to 50:", list(fib_up_to_50))

Output

Fibonacci to 10: [0, 1, 1, 2, 3, 5, 8]
Fibonacci to 50: [0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Practice

Exercise 1

What is the main benefit of using a closure to create configured generators?

Solution

It allows you to enclose configuration values (like limit or multiplier) in the outer scope, keeping the generator function signature clean and modular.

Pattern 3: Decorated Data Pipelines

In real-world data science and machine learning applications, generators are often chained together to process large streams of data. Decorators can be used to monitor metrics (like execution time, memory usage, or record count) across the entire pipeline.

Example: Monitoring a Pipeline

import time

def monitor_step(func):
    def wrapper(data_stream):
        start_time = time.time()
        count = 0
        
        # Process the incoming generator stream
        for item in func(data_stream):
            count += 1
            yield item
            
        end_time = time.time()
        duration = end_time - start_time
        print(f"[Pipeline] {func.__name__} processed {count} items in {duration:.4f}s.")
        
    return wrapper

@monitor_step
def extract_numbers(limit):
    for i in range(1, limit + 1):
        yield i

@monitor_step
def square_numbers(numbers):
    for num in numbers:
        yield num * num

# Run pipeline
pipeline = square_numbers(extract_numbers(5))
print("Pipeline initialized. Fetching values...")
print(list(pipeline))

Output

Pipeline initialized. Fetching values...
[Pipeline] extract_numbers processed 5 items in 0.0000s.
[Pipeline] square_numbers processed 5 items in 0.0001s.
[1, 4, 9, 16, 25]

Check Your Understanding

Question 1 Why does a standard decorator that simply executes a wrapped function not work out-of-the-box for logging yields inside a generator?

Solution

Because a generator function returns a generator object immediately when called instead of running the code block. To log yields, the decorator’s wrapper function must actively iterate over the generator object and yield each item.

Question 2 True or False: Using closures to configure generators helps keep local variables separated across multiple instances.

Solution

True. Each outer function call creates a new scope (closure), ensuring that configuration states and variables do not leak or interfere with other generator instances.

Question 3 How do decorators help when building modular streaming pipelines using generators?

Solution

They allow you to add auxiliary tasks (like performance profiling, error handling, rate limiting, and item auditing) to each pipeline step cleanly without polluting the core data-transformation logic.

Advanced Concepts

Pydantic

Learning Objectives

Pattern 1: Decorating Generators

The Generator Decorator Nuance

Example: Yield Logger

Practice

Exercise 1

Pattern 2: Stateful Iterators with Closures

Example: Configurable Fibonacci Generator

Practice

Exercise 1

Pattern 3: Decorated Data Pipelines

Example: Monitoring a Pipeline

Check Your Understanding

​Learning Objectives

​Pattern 1: Decorating Generators

​The Generator Decorator Nuance

​Example: Yield Logger

​Practice

​Exercise 1

​Pattern 2: Stateful Iterators with Closures

​Example: Configurable Fibonacci Generator

​Practice

​Exercise 1

​Pattern 3: Decorated Data Pipelines

​Example: Monitoring a Pipeline

​Check Your Understanding

Learning Objectives

Pattern 1: Decorating Generators

The Generator Decorator Nuance

Example: Yield Logger

Practice

Exercise 1

Pattern 2: Stateful Iterators with Closures

Example: Configurable Fibonacci Generator

Practice

Exercise 1

Pattern 3: Decorated Data Pipelines

Example: Monitoring a Pipeline

Check Your Understanding