Advanced Python: Iterators, Generators & Decorators

Introduction

After learning Python fundamentals, functions, file handling, exception handling, and object-oriented programming, the next step is to explore some of Python's more advanced features.

These features help you write code that is:

• More Pythonic
• More reusable
• More memory-efficient
• Easier to maintain
• Better suited for larger applications

In this module, you will learn how Python handles iteration, generators, decorators, closures, comprehensions, context managers, dataclasses, specialized collections, and enumerations.

You will also build practical examples such as a custom iterator and a function logging decorator.

Iterators

An iterator is an object that produces values one at a time.

Python's iteration system is based on two important functions:

• iter() — obtains an iterator from an iterable.
• next() — retrieves the next value from an iterator.

Many Python objects are iterable, including:

• Lists
• Tuples
• Strings
• Dictionaries
• Sets
• Files
• Ranges

For example:

pythonCopy
1numbers = [10, 20, 30]
2
3for number in numbers:
4    print(number)

The for loop uses Python's iterator protocol internally.

You can work with the iterator directly:

pythonCopy
1numbers = [10, 20, 30]
2
3iterator = iter(numbers)
4
5print(next(iterator))
6print(next(iterator))
7print(next(iterator))

Output:

textCopy
110
220
330

When there are no more values, calling next() raises StopIteration:

pythonCopy
1next(iterator)

Output:

textCopy
1StopIteration

Usually, you do not need to handle StopIteration manually because Python's for loop handles it automatically.

Iterable vs Iterator

An iterable is an object that can provide an iterator.

An iterator is an object that keeps track of its current position and provides the next value.

For example:

pythonCopy
1numbers = [10, 20, 30]
2
3print(iter(numbers))

The list is iterable, while the object returned by iter(numbers) is an iterator.

A useful way to remember the relationship is:

textCopy
1Iterable
2   │
3   │ iter()
4   ▼
5Iterator
6   │
7   │ next()
8   ▼
9Next value

An iterator implements the iterator protocol through:

pythonCopy
1__iter__()
2__next__()

Creating a Custom Iterator

You can create your own iterator by implementing __iter__() and __next__().

pythonCopy
1class Countdown:
2
3    def __init__(self, start):
4        self.current = start
5
6    def __iter__(self):
7        return self
8
9    def __next__(self):
10        if self.current <= 0:
11            raise StopIteration
12
13        value = self.current
14        self.current -= 1
15
16        return value
17
18
19countdown = Countdown(5)
20
21for number in countdown:
22    print(number)

Output:

textCopy
15
24
33
42
51

The __next__() method must raise StopIteration when iteration is complete.

For simple sequences, generators are often easier to write than custom iterator classes.

Generators

A generator is a convenient way to create an iterator.

Generators use the yield keyword to produce values one at a time.

Example:

pythonCopy
1def numbers():
2    yield 1
3    yield 2
4    yield 3
5
6
7generator = numbers()
8
9print(next(generator))
10print(next(generator))
11print(next(generator))

Output:

textCopy
11
22
33

Unlike return, yield pauses the function while preserving its current state.

When the generator is resumed, execution continues from where it stopped.

Generator with a Loop

Generators are usually consumed with a for loop:

pythonCopy
1def numbers():
2    yield 1
3    yield 2
4    yield 3
5
6
7for number in numbers():
8    print(number)

Output:

textCopy
11
22
33

This is often cleaner than manually calling next().

Generator vs Normal Function

A normal function can return a complete collection:

pythonCopy
1def get_numbers():
2    return [1, 2, 3]

A generator produces values as they are requested:

pythonCopy
1def get_numbers():
2    yield 1
3    yield 2
4    yield 3

The important difference is that a generator does not need to create and store all generated values at once.

Generator for Large Data

Consider generating one million numbers.

A list comprehension creates the complete list:

pythonCopy
1numbers = [x for x in range(1_000_000)]

A generator expression produces values lazily:

pythonCopy
1numbers = (x for x in range(1_000_000))

The generator does not store all one million generated values as a list.

You can process the values one at a time:

pythonCopy
1numbers = (x * 2 for x in range(1_000_000))
2
3for number in numbers:
4    if number > 10:
5        break

Important Note

Generators are not automatically "faster" in every situation. Their main advantage is lazy evaluation and lower memory usage when values do not need to exist simultaneously.

Infinite Generators

Generators can also produce an unlimited sequence of values.

pythonCopy
1def counter():
2    number = 1
3
4    while True:
5        yield number
6        number += 1
7
8
9counter_generator = counter()
10
11for _ in range(5):
12    print(next(counter_generator))

Output:

textCopy
11
22
33
44
55

The generator continues producing values until you stop requesting them.

Decorators

A decorator is a callable that modifies or extends the behavior of another function or class.

Decorators are commonly used for:

• Logging
• Authentication
• Timing
• Caching
• Validation
• Access control

The @decorator_name syntax is shorthand for applying a decorator.

Basic Decorator

pythonCopy
1def greeting_decorator(func):
2
3    def wrapper():
4        print("Before function")
5
6        func()
7
8        print("After function")
9
10    return wrapper
11
12
13@greeting_decorator
14def hello():
15    print("Hello")
16
17
18hello()

Output:

textCopy
1Before function
2Hello
3After function

This:

pythonCopy
1@greeting_decorator
2def hello():
3    print("Hello")

is conceptually equivalent to:

pythonCopy
1def hello():
2    print("Hello")
3
4
5hello = greeting_decorator(hello)

Decorators with Arguments

A decorator should generally use *args and **kwargs when it needs to support functions with different signatures.

pythonCopy
1from functools import wraps
2
3
4def logger(func):
5
6    @wraps(func)
7    def wrapper(*args, **kwargs):
8        print(f"Calling {func.__name__}")
9
10        result = func(*args, **kwargs)
11
12        print(f"{func.__name__} completed")
13
14        return result
15
16    return wrapper
17
18
19@logger
20def greet(name):
21    return f"Hello, {name}!"
22
23
24print(greet("Ankit"))

Output:

textCopy
1Calling greet
2greet completed
3Hello, Ankit!

The @wraps() decorator preserves useful metadata such as the original function's name and docstring.

Decorator for Measuring Execution Time

Decorators are useful for monitoring function performance.

pythonCopy
1from functools import wraps
2from time import perf_counter
3
4
5def measure_time(func):
6
7    @wraps(func)
8    def wrapper(*args, **kwargs):
9        start = perf_counter()
10
11        result = func(*args, **kwargs)
12
13        elapsed = perf_counter() - start
14
15        print(f"{func.__name__} took {elapsed:.6f} seconds")
16
17        return result
18
19    return wrapper
20
21
22@measure_time
23def calculate_sum():
24    return sum(range(1_000_000))
25
26
27print(calculate_sum())

This pattern is useful for debugging and performance analysis.

Closures

A closure is a function that remembers variables from its enclosing scope even after the outer function has finished executing.

Example:

pythonCopy
1def create_greeting(message):
2
3    def greet(name):
4        return f"{message}, {name}!"
5
6    return greet
7
8
9hello = create_greeting("Hello")
10
11print(hello("Ankit"))

Output:

textCopy
1Hello, Ankit!

The greet() function remembers the message value created by create_greeting().

Closure Example: Multiplier

pythonCopy
1def multiplier(factor):
2
3    def multiply(number):
4        return number * factor
5
6    return multiply
7
8
9double = multiplier(2)
10triple = multiplier(3)
11
12print(double(10))
13print(triple(10))

Output:

textCopy
120
230

The double function remembers factor = 2, while triple remembers factor = 3.

Closures are especially useful for creating configurable functions and are closely related to how decorators work.

Comprehensions

Comprehensions provide a concise way to create collections from iterable data.

Python supports:

• List comprehensions
• Dictionary comprehensions
• Set comprehensions
• Generator expressions

List Comprehension

Traditional approach:

pythonCopy
1numbers = []
2
3for number in range(5):
4    numbers.append(number)
5
6print(numbers)

List comprehension:

pythonCopy
1numbers = [number for number in range(5)]
2
3print(numbers)

Output:

textCopy
1[0, 1, 2, 3, 4]

The general structure is:

pythonCopy
1[expression for item in iterable]

List Comprehension with a Condition

You can add a filtering condition.

pythonCopy
1even_numbers = [
2    number
3    for number in range(10)
4    if number % 2 == 0
5]
6
7print(even_numbers)

Output:

textCopy
1[0, 2, 4, 6, 8]

The general structure is:

pythonCopy
1[expression for item in iterable if condition]

Conditional Expression in a Comprehension

You can also use an if-else expression.

pythonCopy
1numbers = [1, 2, 3, 4, 5]
2
3labels = [
4    "Even" if number % 2 == 0 else "Odd"
5    for number in numbers
6]
7
8print(labels)

Output:

textCopy
1['Odd', 'Even', 'Odd', 'Even', 'Odd']

Do not make comprehensions unnecessarily complicated. If the logic becomes difficult to read, use a normal for loop.

Dictionary Comprehension

Dictionary comprehensions create dictionaries concisely.

pythonCopy
1squares = {
2    number: number ** 2
3    for number in range(5)
4}
5
6print(squares)

Output:

textCopy
1{0: 0, 1: 1, 2: 4, 3: 9, 4: 16}

Set Comprehension

Set comprehensions create sets.

pythonCopy
1squares = {
2    number ** 2
3    for number in range(5)
4}
5
6print(squares)

Output:

textCopy
1{0, 1, 4, 9, 16}

Remember that sets do not guarantee a particular display or iteration order.

Generator Expressions

A generator expression looks similar to a list comprehension but uses parentheses.

pythonCopy
1numbers = (
2    number ** 2
3    for number in range(5)
4)
5
6print(next(numbers))
7print(next(numbers))

Output:

textCopy
10
21

The generator expression produces values lazily.

Context Managers

A context manager manages resources and ensures that cleanup happens correctly.

Common resources include:

• Files
• Database connections
• Locks
• Network resources

Python's with statement is the most common way to use a context manager.

File Context Manager

Without a context manager:

pythonCopy
1file = open("data.txt", "r", encoding="utf-8")
2
3try:
4    content = file.read()
5finally:
6    file.close()

With a context manager:

pythonCopy
1with open("data.txt", "r", encoding="utf-8") as file:
2    content = file.read()
3
4print(content)

The with statement ensures that the file is properly closed after the block finishes, including when an exception occurs.

Custom Context Manager

A class can implement the context manager protocol using:

• __enter__()
• __exit__()

Example:

pythonCopy
1class DatabaseConnection:
2
3    def __enter__(self):
4        print("Connected")
5        return self
6
7    def __exit__(self, exc_type, exc_value, traceback):
8        print("Disconnected")
9        return False
10
11
12with DatabaseConnection() as database:
13    print("Working with database...")

Output:

textCopy
1Connected
2Working with database...
3Disconnected

Returning False from __exit__() means exceptions are not suppressed.

Context Managers with contextlib

For simple resource-management logic, contextlib.contextmanager can be easier than writing a class.

pythonCopy
1from contextlib import contextmanager
2
3
4@contextmanager
5def database_connection():
6    print("Connected")
7
8    try:
9        yield
10    finally:
11        print("Disconnected")
12
13
14with database_connection():
15    print("Running database operation")

Output:

textCopy
1Connected
2Running database operation
3Disconnected

The finally block guarantees cleanup.

Dataclasses

The dataclasses module provides a convenient way to create classes that primarily store data.

Without a dataclass:

pythonCopy
1class Student:
2
3    def __init__(self, name, age):
4        self.name = name
5        self.age = age

With a dataclass:

pythonCopy
1from dataclasses import dataclass
2
3
4@dataclass
5class Student:
6    name: str
7    age: int
8
9
10student = Student("Ankit", 23)
11
12print(student)

Output:

textCopy
1Student(name='Ankit', age=23)

A dataclass can automatically provide useful methods such as __init__() and __repr__().

Dataclass with Default Values

pythonCopy
1from dataclasses import dataclass
2
3
4@dataclass
5class Employee:
6    name: str
7    salary: int = 30_000
8
9
10employee = Employee("Rahul")
11
12print(employee)

Output:

textCopy
1Employee(name='Rahul', salary=30000)

Dataclass with Methods

Dataclasses are not limited to storing data. They can also contain methods.

pythonCopy
1from dataclasses import dataclass
2
3
4@dataclass
5class Rectangle:
6    width: float
7    height: float
8
9    def area(self):
10        return self.width * self.height
11
12
13rectangle = Rectangle(10, 5)
14
15print(rectangle.area())

Output:

textCopy
150

NamedTuple

NamedTuple provides tuple-like objects with named fields.

pythonCopy
1from typing import NamedTuple
2
3
4class Student(NamedTuple):
5    name: str
6    age: int
7
8
9student = Student("Ankit", 23)
10
11print(student.name)
12print(student.age)

Output:

textCopy
1Ankit
223

A NamedTuple is immutable, meaning its fields cannot normally be changed after creation.

pythonCopy
1# student.age = 24

This raises an error.

For modern Python applications, consider whether a dataclass or regular class is more appropriate when you need mutable or behavior-rich objects.

The collections Module

The collections module provides specialized container types that solve common data-processing problems.

Some useful types include:

• Counter
• defaultdict
• deque
• OrderedDict

Counter

Counter counts how many times each value occurs.

pythonCopy
1from collections import Counter
2
3
4text = "banana"
5
6counts = Counter(text)
7
8print(counts)

Output:

textCopy
1Counter({'a': 3, 'n': 2, 'b': 1})

You can also find the most common values:

pythonCopy
1print(counts.most_common(2))

Output:

textCopy
1[('a', 3), ('n', 2)]

This is useful for word frequencies, character counts, statistics, and analytics.

defaultdict

defaultdict automatically creates a default value when a missing key is accessed.

pythonCopy
1from collections import defaultdict
2
3
4students = defaultdict(int)
5
6students["Ankit"] += 1
7students["Rahul"] += 1
8
9print(students)

Output:

textCopy
1defaultdict(<class 'int'>, {'Ankit': 1, 'Rahul': 1})

Without defaultdict, you would need to initialize the keys manually.

defaultdict for Grouping

A common practical use is grouping values.

pythonCopy
1from collections import defaultdict
2
3
4students_by_course = defaultdict(list)
5
6students_by_course["Python"].append("Ankit")
7students_by_course["Python"].append("Rahul")
8students_by_course["Java"].append("Priya")
9
10print(dict(students_by_course))

Output:

textCopy
1{'Python': ['Ankit', 'Rahul'], 'Java': ['Priya']}

deque

deque stands for double-ended queue.

It provides efficient operations at both ends.

pythonCopy
1from collections import deque
2
3
4queue = deque()
5
6queue.append("Ankit")
7queue.append("Rahul")
8
9print(queue)
10
11queue.popleft()
12
13print(queue)

Output:

textCopy
1deque(['Ankit', 'Rahul'])
2deque(['Rahul'])

You can also add or remove items from the left:

pythonCopy
1queue.appendleft("Priya")
2queue.pop()

A deque is often a better choice than a list when you frequently add or remove elements from both ends.

OrderedDict

Modern Python dictionaries preserve insertion order, so OrderedDict is no longer necessary simply to maintain dictionary order.

However, OrderedDict still provides specialized behavior, such as moving keys.

pythonCopy
1from collections import OrderedDict
2
3
4data = OrderedDict()
5
6data["A"] = 1
7data["B"] = 2
8data["C"] = 3
9
10data.move_to_end("A")
11
12print(data)

Output:

textCopy
1OrderedDict([('B', 2), ('C', 3), ('A', 1)])

For normal ordered mappings, the built-in dict is usually sufficient.

Enumerations with Enum

An enum represents a fixed set of named values.

Enums are useful for states, categories, permissions, directions, and other predefined choices.

pythonCopy
1from enum import Enum
2
3
4class Status(Enum):
5    PENDING = 1
6    RUNNING = 2
7    COMPLETED = 3
8
9
10print(Status.PENDING)
11print(Status.PENDING.name)
12print(Status.PENDING.value)

Output:

textCopy
1Status.PENDING
2PENDING
31

Iterating Through an Enum

You can loop through enum members:

pythonCopy
1for status in Status:
2    print(status.name, status.value)

Output:

textCopy
1PENDING 1
2RUNNING 2
3COMPLETED 3

Enums make code easier to understand than using unexplained numeric constants.

String Enums

When values are intended to be strings, StrEnum is available in modern Python versions.

pythonCopy
1from enum import StrEnum
2
3
4class Environment(StrEnum):
5    DEVELOPMENT = "development"
6    TESTING = "testing"
7    PRODUCTION = "production"
8
9
10print(Environment.PRODUCTION)

This can be useful when application configuration uses predefined string values.

Practice Project: Custom Countdown Iterator

Build a reusable countdown iterator.

pythonCopy
1class Countdown:
2
3    def __init__(self, start):
4        if start < 0:
5            raise ValueError("Start value cannot be negative")
6
7        self.current = start
8
9    def __iter__(self):
10        return self
11
12    def __next__(self):
13        if self.current == 0:
14            raise StopIteration
15
16        value = self.current
17        self.current -= 1
18
19        return value
20
21
22countdown = Countdown(5)
23
24for number in countdown:
25    print(number)

Output:

textCopy
15
24
33
42
51

This project demonstrates:

• Classes
• Object state
• __iter__()
• __next__()
• StopIteration
• The iterator protocol
• Input validation

Challenge

Modify the iterator so that it counts down by a configurable step:

pythonCopy
1Countdown(10, step=2)

Expected output:

textCopy
110
28
36
44
52

Practice Project: Function Logging Decorator

Decorators are commonly used to add logging without changing the original function.

pythonCopy
1from functools import wraps
2
3
4def log_function(func):
5
6    @wraps(func)
7    def wrapper(*args, **kwargs):
8        print(f"Calling {func.__name__}")
9
10        result = func(*args, **kwargs)
11
12        print(f"{func.__name__} finished")
13
14        return result
15
16    return wrapper
17
18
19@log_function
20def add(a, b):
21    return a + b
22
23
24result = add(10, 20)
25
26print(result)

Output:

textCopy
1Calling add
2add finished
330

Challenge

Modify the decorator so that it also prints the returned result:

textCopy
1Calling add
2add returned 30
3add finished

Choosing the Right Feature

Different advanced Python features solve different problems.

Common Mistakes

Calling next() on an Exhausted Iterator

pythonCopy
1numbers = iter([1, 2])
2
3print(next(numbers))
4print(next(numbers))
5
6# next(numbers)

After the available values have been consumed, StopIteration is raised.

Forgetting to Return the Result in a Decorator

Incorrect:

pythonCopy
1def decorator(func):
2
3    def wrapper(*args, **kwargs):
4        func(*args, **kwargs)
5
6    return wrapper

If the original function returns a value, the wrapper should normally return it too:

pythonCopy
1def decorator(func):
2
3    def wrapper(*args, **kwargs):
4        result = func(*args, **kwargs)
5        return result
6
7    return wrapper

Forgetting @wraps

A decorator that replaces a function can hide metadata from the original function.

Prefer:

pythonCopy
1from functools import wraps

and:

pythonCopy
1@wraps(func)

when writing general-purpose decorators.

Making Comprehensions Too Complex

This can become difficult to read:

pythonCopy
1result = [
2    x * 2
3    for x in numbers
4    if x > 10
5    if x % 2 == 0
6]

A comprehension is useful when the logic remains readable. For complicated transformations, a normal loop or helper function may be better.

Assuming Generators Can Be Reused

Generators are consumed as you iterate over them.

pythonCopy
1numbers = (x for x in range(3))
2
3print(list(numbers))
4print(list(numbers))

Output:

textCopy
1[0, 1, 2]
2[]

Once the generator is exhausted, it does not automatically restart.

Forgetting Resource Cleanup

Avoid manually managing resources when a context manager is available.

Prefer:

pythonCopy
1with open("data.txt", encoding="utf-8") as file:
2    content = file.read()

instead of relying on a manually called close().

Practice Exercises

Exercise 1: Even Number Generator

Create a generator that produces even numbers from 2 through 20.

Expected output:

textCopy
12
24
36
48
510
612
714
816
918
1020

Exercise 2: Fibonacci Generator

Create a generator that produces the first 10 Fibonacci numbers.

Exercise 3: Timing Decorator

Create a decorator that measures and prints how long a function takes to execute.

Exercise 4: Password Validation Decorator

Create a decorator that checks whether a function receives a non-empty password before executing.

Exercise 5: Student Dataclass

Create a dataclass named Student with:

• name
• age
• marks

Add a method called is_passed() that returns True when marks are at least 40.

Exercise 6: Word Counter

Use Counter to count the frequency of words in:

textCopy
1python is easy and python is powerful

Exercise 7: Queue

Use deque to create a queue that supports:

• Add customer
• Serve customer
• Display waiting customers

Exercise 8: Application Status

Create an enum containing:

• PENDING
• PROCESSING
• SUCCESS
• FAILED

Use it in a small application-status example.

Key Takeaways

• An iterable can provide an iterator.
• An iterator produces values one at a time through next().
• Generators provide a simple way to create iterators using yield.
• Generators are useful for lazy processing and memory-efficient pipelines.
• Decorators extend or modify function behavior without changing the original function's source code.
• functools.wraps() preserves important function metadata when writing decorators.
• Closures remember values from their enclosing scope.
• Comprehensions provide concise collection-building syntax.
• Generator expressions provide lazy evaluation.
• Context managers handle resource setup and cleanup safely.
• dataclass simplifies data-focused classes.
• NamedTuple provides immutable, tuple-like records with named fields.
• Counter, defaultdict, and deque solve common collection-processing problems.
• Enum makes fixed sets of named values easier to understand and maintain.
• Advanced Python features should improve readability and design rather than make code unnecessarily complicated.

The goal of advanced Python is not to use every feature everywhere. The goal is to understand when a feature makes your code clearer, safer, more reusable, or more efficient.