Python Tutorial / Process Data Lazily with Iterators and Generators

An iterator provides one next item at a time; a generator lets you write that behavior with yield. Values can be calculated on demand, but an exhausted iterator does not automatically restart.

Separate an iterable from its iterator

A list is iterable; iter(the_list) creates an iterator over it. next() requests a value and raises StopIteration when none remain. A for loop handles that termination signal for you.

Calling a function containing yield creates a generator rather than running its body to completion. Requesting the next value advances execution; yield pauses it while retaining the local state needed to continue.

For the relevant APIs and platform requirements, see Python: classes.

File streams and consecutive grouping build on the rules demonstrated in Aggregate and Combine Iteration with collections and itertools.

Consume running totals step by step

Save the complete code below as generators.py in a separate practice folder, using UTF-8. Run python generators.py in a terminal with your project virtual environment selected. Outside a virtual environment on macOS or Ubuntu, use python3 when that is your interpreter command.

def running_totals(amounts):
    total = 0
    for amount in amounts:
        total += amount
        yield total

values = running_totals([3500, 8000, 2500])
print(next(values))
print(list(values))
print(list(values))
print(list(running_totals([3500, 8000, 2500])))

Expected output:

3500
[11500, 14000]
[]
[3500, 11500, 14000]

Evaluate the real memory and timing tradeoff

Generating each cumulative result on request avoids storing the whole output in advance. An input already held in a large list still consumes its memory, and list(generator) collects all generated results into memory again.

A generator expression has the form (expression for item in source). A small result needed repeatedly may be clearer as a list. Lazy evaluation changes when computation and storage happen; it does not guarantee that every workload becomes faster.

A list stores values; iter(list) stores a current traversal position over those values. After one next call, collecting the remainder does not repeat the consumed item. A second list conversion of the exhausted iterator is empty, while the original list remains. The generator expression below is also consumed and computes values on demand.

Save the complete example as iterator.py and run python iterator.py in the same practice directory.

values = [1, 2, 3]
iterator = iter(values)
print(next(iterator))
print(list(iterator))
print(list(iterator))
print(list(values))
expression = (n * 2 for n in values)
print(list(expression))

Expected output:

1
[2, 3]
[]
[1, 2, 3]
[2, 4, 6]

Handle delayed errors and resource lifetime

An exception inside a generator body can occur while consuming values rather than when creating the generator. If a generator opens a file, decide how the file closes when consumption stops early. A caller-controlled with block can make resource lifetime easier to reason about.

1. Call next one additional time and inspect the smaller remaining sequence.
2. Use an empty input and observe StopIteration on the first next call.
3. Create a fresh generator or store results when you need to read them twice.

The file lifetime lesson provides the resource-management context for generators that process file content.