ValueError: The truth value of an array with more

Question

I just discovered a logical bug in my code which was causing all sorts of problems. I was inadvertently doing a bitwise AND instead of a logical AND.

I changed the code from:

r = mlab.csv2rec(datafile, delimiter=',', names=COL_HEADERS)
mask = ((r["dt"] >= startdate) & (r["dt"] <= enddate))
selected = r[mask]

TO:

r = mlab.csv2rec(datafile, delimiter=',', names=COL_HEADERS)
mask = ((r["dt"] >= startdate) and (r["dt"] <= enddate))
selected = r[mask]

To my surprise, I got the rather cryptic error message:

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

Why was a similar error not emitted when I use a bitwise operation - and how do I fix this?

Answer 1

r is a numpy (rec)array. So r["dt"] >= startdate is also a (boolean) array. For numpy arrays the & operation returns the bitwise-and of the two boolean arrays.

The NumPy developers felt there was no one commonly understood way to evaluate an array in boolean context: it could mean True if any element is True, or it could mean True if all elements are True, or True if the array has non-zero length, just to name three possibilities.

Since different users might have different needs and different assumptions, the NumPy developers refused to guess and instead decided to raise a ValueError whenever one tries to evaluate an array in boolean context. Applying and to two numpy arrays causes the two arrays to be evaluated in boolean context (by calling __bool__ in Python3 or __nonzero__ in Python2).

Your original code

mask = ((r["dt"] >= startdate) & (r["dt"] <= enddate))
selected = r[mask]

looks correct. However, if you do want and, then instead of a and b use (a-b).any() or (a-b).all().

Answer 2

I had the same problem (i.e. indexing with multi-conditions, here it's finding data in a certain date range). The (a-b).any() or (a-b).all() seem not working, at least for me.

Alternatively I found another solution which works perfectly for my desired functionality (https://stackoverflow.com/questions/12647471/the-truth-value-of-an-array-with-more-than-one-element-is-ambigous-when-trying-t).

Instead of using suggested code above, simply using a numpy.logical_and(a,b) would work. Here you may want to rewrite the code as

selected = r(logical_and(r["dt"] >= startdate, r["dt"] <= enddate))

Answer 3

The reason for the exception is that and implicitly calls bool. First on the left operand and (if the left operand is True) then on the right operand. So x and y is equivalent to bool(x) and bool(y).

However the bool on a numpy.ndarray (if it contains more than one element) will throw the exception you have seen:

>>> import numpy as np
>>> arr = np.array([1, 2, 3])
>>> bool(arr)
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

The bool() call is implicit in and, but also in if, while, or, so any of the following examples will also fail:

>>> arr and arr
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

>>> if arr: pass
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

>>> while arr: pass
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

>>> arr or arr
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

There are more functions and statements in Python that hide bool calls, for example 2 < x < 10 is just another way of writing 2 < x and x < 10. And the and will call bool: bool(2 < x) and bool(x < 10).

The element-wise equivalent for and would be the np.logical_and function, similarly you could use np.logical_or as equivalent for or.

For boolean arrays - and comparisons like <, <=, ==, !=, >= and > on NumPy arrays return boolean NumPy arrays - you can also use the element-wise bitwise functions (and operators): np.bitwise_and (& operator)

>>> np.logical_and(arr > 1, arr < 3)
array([False,  True, False], dtype=bool)

>>> np.bitwise_and(arr > 1, arr < 3)
array([False,  True, False], dtype=bool)

>>> (arr > 1) & (arr < 3)
array([False,  True, False], dtype=bool)

and bitwise_or (| operator):

>>> np.logical_or(arr <= 1, arr >= 3)
array([ True, False,  True], dtype=bool)

>>> np.bitwise_or(arr <= 1, arr >= 3)
array([ True, False,  True], dtype=bool)

>>> (arr <= 1) | (arr >= 3)
array([ True, False,  True], dtype=bool)

A complete list of logical and binary functions can be found in the NumPy documentation:

• "Logic functions"
• "Binary operations"