exhausted iterators - what to do about them?

Question

(In Python 3.1) (Somewhat related to another question I asked, but this question is about iterators being exhausted.)

# trying to see the ratio of the max and min element in a container c
filtered = filter(lambda x : x is not None and x != 0, c)
ratio = max(filtered) / min(filtered)

It took me half hour to realize what the problem is (the iterator returned by filter is exhausted by the time it gets to the second function call). How do I rewrite it in the most Pythonic / canonical way?

Also, what can I do to avoid bugs of this sort, besides getting more experience? (Frankly, I don't like this language feature, since these types of bugs are easy to make and hard to catch.)

Answer 1

The itertools.tee function can help here:

import itertools

f1, f2 = itertools.tee(filtered, 2)
ratio = max(f1) / min(f2)

Answer 2

The entity filtered is essentially an object with state. Of course, now it's obivous that running max or min on it will change that state. In order to stop tripping over that, I like to make it absolutely clear (to myself, really) that I am constructing something rather than just transforming something:

Adding an extra step can really help:

def filtered(container):
    return filter(lambda x : x is not None and x != 0, container)

ratio = max(filtered(c)) / min(filtered(c))

Whether you put filtered(...) inside some function (maybe it's not really needed for anything else) or define it as a module-level function is up to you, but in this case I'd suggest that if filtered (the iterator) was only needed in the function, leave it there until you need it elsewhere.

The other thing you can do is to construct a list from it, which will evaluate the iterator:

filtered_iter = filter(lambda x : x is not None and x != 0, container)
filtered = list(filtered_iter)

ratio = max(filtered) / min(filtered)

(Of course, you can just say filtered = list(filter(...)).)

Answer 3

Actually your code raises an exception that would prevent this problem! So I guess the problem was that you masked the exception?

>>> min([])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: min() arg is an empty sequence
>>> min(x for x in ())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: min() arg is an empty sequence

Anyways, you can also write a new function to give you the min and max at the same time:

def minmax( seq ):
    " returns the `(min, max)` of sequence `seq`"
    it = iter(seq)
    try:
        min = max = next(it)
    except StopIteration:
        raise ValueError('arg is an empty sequence')
    for item in it:
        if item < min:
            min = item
        elif item > max:
            max = item
    return min, max

Answer 4

you can convert an iterator to a tuple simply by calling tuple(iterator)

however I'd rewrite that filter as a list comprehension, which would look something like this

# original
filtered = filter(lambda x : x is not None and x != 0, c)

# list comp
filtered = [x for x in c if x is not None and x != 0]