Python doctest execution context

问题:

I have the following function which I do unit testing with doctest.

from collections import deque

def fill_q(histq=deque([])):
    """
    >>> fill_q()
    deque([1, 2, 3])
    >>> fill_q()
    deque([1, 2, 3])
    """
    if histq:
        assert(len(histq) == 0)
    histq.append(1)
    histq.append(2)
    histq.append(3)
    return histq

if __name__ == "__main__":
    import doctest
    doctest.testmod()

the first case passes, but the second call to fill_q fails, yet it's the same code:

**********************************************************************
File "trial.py", line 7, in __main__.fill_q
Failed example:
    fill_q()
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python2.7/doctest.py", line 1289, in __run
        compileflags, 1) in test.globs
      File "<doctest __main__.fill_q[1]>", line 1, in <module>
        fill_q()
      File "trial.py", line 11, in fill_q
        assert(len(histq) == 0)
    AssertionError
**********************************************************************
1 items had failures:
   1 of   2 in __main__.fill_q
***Test Failed*** 1 failures.

It looks like that doctest re-uses the local variable histq from the first test call, why is it doing this? This is very silly behaviour (provided it's not me doing sth crazy here).

回答1:

The problem is not with doctest, but the default parameter you are using in def fill_q(histq=deque([])). It is similar to this:

>>> from collections import deque
>>> 
>>> def fill_q(data=deque([])):
...     data.append(1)
...     return data
... 
>>> fill_q()
deque([1])
>>> fill_q()
deque([1, 1])
>>> fill_q()
deque([1, 1, 1])

This seemingly odd behaviour happens when you use a mutable object as a default value like a list or a dictionary. It is in fact using the same object:

>>> id(fill_q())
4485636624
>>> id(fill_q())
4485636624
>>> id(fill_q())
4485636624

---

Why?

Default parameter values are always evaluated when and only when the def statement they belong to is executed [ref].

---

How to avoid this mistake:

Use None as default parameter instead, or for arbitrary object:

my_obj = object()
def sample_func(value=my_obj):
    if value is my_obj:
        value = expression
    # then modify value 

---

When to use it?:

1. local rebinding of global names:

   import math
   
   def fast_func(sin=math.sin, cos=math.cos):
   

2. can be used for memoization (e.g., make certain recursions run faster)

回答2:

You're making a very common Python mistake - if you set an object to be a default constructor for a function, it will not be reinitialized on the next invocation of that function - and any changes to that object will persist across function calls.

A better strategy that avoids this problem is to set the default to some known value, and check for it:

def fill_q(histq=None):
    if histq is None:
        histq = deque([])
    ...