Iterators & comprehensions

This page covers the tools Python gives you for working through a sequence of values one at a time: list, dictionary and set comprehensions; iter and next; enumerate, zip, map, filter, sorted and reversed; generator expressions; and classes that define their own iteration. In d/Python these are lazy wherever Python makes them lazy: an iterator remembers where it is between calls and pulls one item at a time, while a comprehension builds its whole collection, of one element type, in a single pass. After this page you will know which of these work and which do not yet.

Comprehensions

squares = [n * n for n in range(8) if n % 2 == 0]
pairs = [(a, b) for a in range(3) for b in range(a)]
lengths = {word: len(word) for word in ["fig", "pear"]}
letters = {c for c in "banana"}
print(squares, pairs, lengths, sorted(letters))
matrix = [[r * 3 + c for c in range(3)] for r in range(2)]
print(matrix, [cell for row in matrix for cell in row])

Lazy iterators keep their place

cursor = iter([10, 20, 30])
print(next(cursor), next(cursor))
print(next(cursor), next(cursor, 0))
for index, value in enumerate("ab", start=1):
    print(index, value)
print(list(zip([1, 2, 3], "xy")), list(reversed([1, 2, 3])))
letters = iter("abc")
print(list(zip(letters, letters)))

Pass the same iterator to zip twice and both positions draw from the same place, so letters above pairs a with b. next raises StopIteration when the items run out, unless you give it a default of the same type, which it returns instead. reversed on a list sees changes made to the list while you are still iterating, and once it has run out it stays empty.

map and filter

print(list(map(abs, [-3, 0, 4])))
print(list(map(lambda a, b: a + b, [1, 2, 3], [10, 20])))
print(list(filter(None, [0, 1, 2, 0])), list(filter(lambda n: n % 2, range(7))))

class Threshold:
    def __init__(self, limit):
        self.limit = limit
    def __call__(self, value):
        return value > self.limit

print(list(filter(Threshold(2), [1, 2, 3, 4])))

map evaluates its arguments once, takes hold of its input sequences from left to right, and calls your function only when the next item is asked for; it stops when the shortest input runs out. filter hands back the original items that pass the test; filter(None, ...) keeps the items that count as true. The function you pass to either may be a named function, a lambda, a closure, a method taken from an object, or an object with a __call__ method like Threshold above.

sorted and reversed

words = ["pear", "Fig", "apple", "kiwi"]
print(sorted(words), sorted(words, key=len), sorted(words, key=len, reverse=True))
pairs = [(2, "b"), (1, "z"), (2, "a")]
print(sorted(pairs), sorted(pairs, key=lambda pair: pair[1]))
numbers = [3, 1, 2]
numbers.sort(reverse=True)
print(numbers)

sorted copies its input first, calls the key function once per item in the original order, and keeps equal items in their original order (it is stable), even with reverse=True. A key may be a single value or a tuple of them; the key function may be a named function, a closure, a method taken from an object, or a callable object. Sorting by float keys, and by keys whose types differ from item to item, is not yet supported.

Generator expressions

total = sum(n * n for n in range(10))
print(total, any(n > 8 for n in range(10)), all(n >= 0 for n in range(10)))
flat = (cell for row in [[1, 2], [3]] for cell in row if cell != 2)
print(next(flat), list(flat))

A generator expression may have one or more for clauses, each with an if filter. As in Python, the outermost sequence is evaluated the moment the expression is created, while each inner sequence is evaluated only after the loop variable outside it has a value and its filters have passed. Generator functions (a def containing yield) and the send, throw and close methods are not yet supported; the compatibility table tracks them.

Class-defined iterators

class Countdown:
    def __init__(self, start):
        self.current = start
    def __iter__(self):
        return self
    def __next__(self):
        if self.current == 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

print(list(Countdown(3)), sum(Countdown(4)), [n for n in Countdown(2)])

A class with __iter__ and __next__ works everywhere a built-in iterator does, including list(), sum(), comprehensions and for, and the two methods may be inherited from a parent class. The object you get back from iter() is the same object your __iter__ returned. Not yet supported: looping over a class that defines only __getitem__, the two-argument form iter(function, sentinel), and __length_hint__.

What the tests cover

Every claim on this page comes from a test program that runs in both d/Python and CPython and must print the same thing. The corpus page lists those programs, with their reviewed output.