What exactly are iterator, iterable, and iteration

Question

What are the most basic definitions of "iterable", "iterator" and "iteration in Python?

I've read multiple definitions but their exact meaning still won't sink in.

Can someone please help me with the basic idea?

Answer 1

The above answers are great, but as most of what I've seen, don't stress the distinction enough for people like me.

Also, people tend to get "too Pythonic" by putting definitions like "X is an object that has __foo__() method" before. Such definitions are correct--they are based on duck-typing philosophy, but the focus on methods tends to get between when trying to understand the concept in its simplicity.

So I add my version.

---

In natural language,

• iteration is the process of taking one element at a time in a row of elements.

In Python,

• iterable is an object that is, well, iterable, which simply put, means that it can be used in iteration, e.g. with a for loop. How? By using iterator. I'll explain below.

• ... while iterator is an object that defines how to actually do the iteration--specifically what is the next element. That's why it must have next() method.

Iterators are themselves also iterable, with the distinction that their __iter__() method returns the same object (self), regardless of whether or not its items have been consumed by previous calls to next().

---

So what does Python interpreter think when it sees for x in obj: statement?

Look, a for loop. Looks like a job for an iterator... Let's get one. ... There's this obj guy, so let's ask him.

"Mr. obj, do you have your iterator?" (... calls iter(obj), which calls obj.__iter__(), which happily hands out a shiny new iterator _i.)

OK, that was easy... Let's start iterating then. (x = _i.next() ... x = _i.next()...)

Since Mr. obj succeeded in this test (by having certain method returning a valid iterator), we reward him with adjective: you can now call him "iterable Mr. obj".

However, in simple cases, you don't normally benefit from having iterator and iterable separately. So you define only one object, which is also its own iterator. (Python does not really care that _i handed out by obj wasn't all that shiny, but just the obj itself.)

This is why in most examples I've seen (and what had been confusing me over and over), you can see:

class IterableExample(object):

    def __iter__(self):
        return self

    def next(self):
        pass

instead of

class Iterator(object):
    def next(self):
        pass

class Iterable(object):
    def __iter__(self):
        return Iterator()

There are cases, though, when you can benefit from having iterator separated from the iterable, such as when you want to have one row of items, but more "cursors". For example when you want to work with "current" and "forthcoming" elements, you can have separate iterators for both. Or multiple threads pulling from a huge list: each can have its own iterator to traverse over all items. See @Raymond's and @glglgl's answers above.

Imagine what you could do:

class SmartIterableExample(object):

    def create_iterator(self):
        # An amazingly powerful yet simple way to create arbitrary
        # iterator, utilizing object state (or not, if you are fan
        # of functional), magic and nuclear waste--no kittens hurt.
        pass    # don't forget to add the next() method

    def __iter__(self):
        return self.create_iterator()

Notes:

• I'll repeat again: iterator is not iterable. Iterator cannot be used as a "source" in for loop. What for loop primarily needs is __iter__() (that returns something with next()).

• Of course, for is not the only iteration loop, so above applies to some other constructs as well (while...).

• Iterator's next() can throw StopIteration to stop iteration. Does not have to, though, it can iterate forever or use other means.

• In the above "thought process", _i does not really exist. I've made up that name.

• There's a small change in Python 3.x: next() method (not the built-in) now must be called __next__(). Yes, it should have been like that all along.

• You can also think of it like this: iterable has the data, iterator pulls the next item

Disclaimer: I'm not a developer of any Python interpreter, so I don't really know what the interpreter "thinks". The musings above are solely demonstration of how I understand the topic from other explanations, experiments and real-life experience of a Python newbie.

Answer 2

In Python everything is an object. When an object is said to be iterable, it means that you can step through (i.e. iterate) the object as a collection.

Arrays for example are iterable. You can step through them with a for loop, and go from index 0 to index n, n being the length of the array object minus 1.

Dictionaries (pairs of key/value, also called associative arrays) are also iterable. You can step through their keys.

Obviously the objects which are not collections are not iterable. A bool object for example only have one value, True or False. It is not iterable (it wouldn't make sense that it's an iterable object).

Read more. http://www.lepus.org.uk/ref/companion/Iterator.xml

Answer 3

I don't think that you can get it much simpler than the documentation, however I'll try:

• Iterable is something that can be iterated over. In practice it usually means a sequence e.g. something that has a beginning and an end and some way to go through all the items in it.

• You can think Iterator as a helper pseudo-method (or pseudo-attribute) that gives (or holds) the next (or first) item in the iterable. (In practice it is just an object that defines the method next())

• Iteration is probably best explained by the Merriam-Webster definition of the word :

b : the repetition of a sequence of computer instructions a specified number of times or until a condition is met — compare recursion

Answer 4

I don’t know if it helps anybody but I always like to visualize concepts in my head to better understand them. So as I have a little son I visualize iterable/iterator concept with bricks and white paper.

Suppose we are in the dark room and on the floor we have bricks for my son. Bricks of different size, color, does not matter now. Suppose we have 5 bricks like those. Those 5 bricks can be described as an object – let’s say bricks kit. We can do many things with this bricks kit – can take one and then take second and then third, can change places of bricks, put first brick above the second. We can do many sorts of things with those. Therefore this bricks kit is an iterable object or sequence as we can go through each brick and do something with it. We can only do it like my little son – we can play with one brick at a time. So again I imagine myself this bricks kit to be an iterable.

Now remember that we are in the dark room. Or almost dark. The thing is that we don’t clearly see those bricks, what color they are, what shape etc. So even if we want to do something with them – aka iterate through them – we don’t really know what and how because it is too dark.

What we can do is near to first brick – as element of a bricks kit – we can put a piece of white fluorescent paper in order for us to see where the first brick-element is. And each time we take a brick from a kit, we replace the white piece of paper to a next brick in order to be able to see that in the dark room. This white piece of paper is nothing more than an iterator. It is an object as well. But an object with what we can work and play with elements of our iterable object – bricks kit.

That by the way explains my early mistake when I tried the following in an IDLE and got a TypeError:

 >>> X = [1,2,3,4,5]
 >>> next(X)
 Traceback (most recent call last):
    File "<pyshell#19>", line 1, in <module>
      next(X)
 TypeError: 'list' object is not an iterator

List X here was our bricks kit but NOT a white piece of paper. I needed to find an iterator first:

>>> X = [1,2,3,4,5]
>>> bricks_kit = [1,2,3,4,5]
>>> white_piece_of_paper = iter(bricks_kit)
>>> next(white_piece_of_paper)
1
>>> next(white_piece_of_paper)
2
>>>

Don’t know if it helps, but it helped me. If someone could confirm/correct visualization of the concept, I would be grateful. It would help me to learn more.

Answer 5

An iterable is a object which has a __iter__() method. It can possibly iterated over several times, such as list()s and tuple()s.

An iterator is the object which iterates. It is returned by an __iter__() method, returns itself via its own __iter__() method and has a next() method (__next__() in 3.x).

Iteration is the process of calling this next() resp. __next__() until it raises StopIteration.

Example:

>>> a = [1, 2, 3] # iterable
>>> b1 = iter(a) # iterator 1
>>> b2 = iter(a) # iterator 2, independent of b1
>>> next(b1)
1
>>> next(b1)
2
>>> next(b2) # start over, as it is the first call to b2
1
>>> next(b1)
3
>>> next(b1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration
>>> b1 = iter(a) # new one, start over
>>> next(b1)
1

Answer 6

Here's my cheat sheet:

 sequence
  +
  |
  v
   def __getitem__(self, index: int):
  +    ...
  |    raise IndexError
  |
  |
  |              def __iter__(self):
  |             +     ...
  |             |     return <iterator>
  |             |
  |             |
  +--> or <-----+        def __next__(self):
       +        |       +    ...
       |        |       |    raise StopIteration
       v        |       |
    iterable    |       |
           +    |       |
           |    |       v
           |    +----> and +-------> iterator
           |                               ^
           v                               |
   iter(<iterable>) +----------------------+
                                           |
   def generator():                        |
  +    yield 1                             |
  |                 generator_expression +-+
  |                                        |
  +-> generator() +-> generator_iterator +-+

Quiz: Do you see how...

• every iterator is an iterable?
• a container object's __iter__() method can be implemented as a generator?
• an iterable plus a __next__ method is not necessarily an iterator?