List comprehension without [ ] in Python

Question

Joining a list:

>>> ''.join([ str(_) for _ in xrange(10) ])
'0123456789'

join must take an iterable.

Apparently, join's argument is [ str(_) for _ in xrange(10) ], and it's a list comprehension.

Look at this:

>>>''.join( str(_) for _ in xrange(10) )
'0123456789'

Now, join's argument is just str(_) for _ in xrange(10), no [], but the result is the same.

Why? Does str(_) for _ in xrange(10) also produce a list or an iterable?

Answer 1

The argument to your second join call is a generator expression. It does produce an iterable.

Answer 2

Your second example uses a generator expression rather than a list comprehension. The difference is that with the list comprehension, a list is completely built and passed to .join(). With the generator expression, items are generated one by one and consumed by .join(). The latter uses less memory and is generally faster.

As it happens, the list constructor will happily consume any iterable, including a generator expression. So:

[str(n) for n in xrange(10)]

is just "syntactic sugar" for:

list(str(n) for n in xrange(10))

In other words, a list comprehension is just a generator expression that is turned into a list.

Answer 3

The other respondents were correct in answering that you had discovered a generator expression (which has a notation similar to list comprehensions but without the surrounding square brackets).

In general, genexps (as they are affectionately known) are more memory efficient and faster than list comprehensions.

HOWEVER, it the case of ''.join(), a list comprehension is both faster and more memory efficient. The reason is that join needs to make two passes over the data, so it actually needs a real list. If you give it one, it can start its work immediately. If you give it a genexp instead, it cannot start work until it builds-up a new list in memory by running the genexp to exhaustion:

~ $ python -m timeit '"".join(str(n) for n in xrange(1000))'
1000 loops, best of 3: 335 usec per loop
~ $ python -m timeit '"".join([str(n) for n in xrange(1000)])'
1000 loops, best of 3: 288 usec per loop

The same result holds when comparing itertools.imap versus map:

~ $ python -m timeit -s'from itertools import imap' '"".join(imap(str, xrange(1000)))'
1000 loops, best of 3: 220 usec per loop
~ $ python -m timeit '"".join(map(str, xrange(1000)))'
1000 loops, best of 3: 212 usec per loop

Answer 4

As mentioned it's a generator expression.

From the documentation:

The parentheses can be omitted on calls with only one argument. See section Calls for the detail.

Answer 5

That's a generator, rather than a list comprehension. Generators are also iterables, but rather than creating the entire list first then passing it to join, it passes each value in the xrange one by one, which can be much more efficient.

Answer 6

>>>''.join( str(_) for _ in xrange(10) )

This is called a generator expression, and is explained in PEP 289.

The main difference between generator expressions and list comprehensions is that the former don't create the list in memory.

Note that there's a third way to write the expression:

''.join(map(str, xrange(10)))