Mocking ImportError in Python

Question

I'm trying this for almost two hours now, without any luck.

I have a module that looks like this:

try:
    from zope.component import queryUtility  # and things like this
except ImportError:
    # do some fallback operations <-- how to test this?

Later in the code:

try:
    queryUtility(foo)
except NameError:
    # do some fallback actions <-- this one is easy with mocking 
    # zope.component.queryUtility to raise a NameError

Any ideas?

EDIT:

Alex's suggestion doesn't seem to work:

>>> import __builtin__
>>> realimport = __builtin__.__import__
>>> def fakeimport(name, *args, **kw):
...     if name == 'zope.component':
...         raise ImportError
...     realimport(name, *args, **kw)
...
>>> __builtin__.__import__ = fakeimport

When running the tests:

aatiis@aiur ~/work/ao.shorturl $ ./bin/test --coverage .
Running zope.testing.testrunner.layer.UnitTests tests:
  Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds.


Error in test /home/aatiis/work/ao.shorturl/src/ao/shorturl/shorturl.txt
Traceback (most recent call last):
  File "/usr/lib64/python2.5/unittest.py", line 260, in run
    testMethod()
  File "/usr/lib64/python2.5/doctest.py", line 2123, in runTest
    test, out=new.write, clear_globs=False)
  File "/usr/lib64/python2.5/doctest.py", line 1361, in run
    return self.__run(test, compileflags, out)
  File "/usr/lib64/python2.5/doctest.py", line 1282, in __run
    exc_info)
  File "/usr/lib64/python2.5/doctest.py", line 1148, in report_unexpected_exception
    'Exception raised:\n' + _indent(_exception_traceback(exc_info)))
  File "/usr/lib64/python2.5/doctest.py", line 1163, in _failure_header
    out.append(_indent(source))
  File "/usr/lib64/python2.5/doctest.py", line 224, in _indent
    return re.sub('(?m)^(?!$)', indent*' ', s)
  File "/usr/lib64/python2.5/re.py", line 150, in sub
    return _compile(pattern, 0).sub(repl, string, count)
  File "/usr/lib64/python2.5/re.py", line 239, in _compile
    p = sre_compile.compile(pattern, flags)
  File "/usr/lib64/python2.5/sre_compile.py", line 507, in compile
    p = sre_parse.parse(p, flags)
AttributeError: 'NoneType' object has no attribute 'parse'



Error in test BaseShortUrlHandler (ao.shorturl)
Traceback (most recent call last):
  File "/usr/lib64/python2.5/unittest.py", line 260, in run
    testMethod()
  File "/usr/lib64/python2.5/doctest.py", line 2123, in runTest
    test, out=new.write, clear_globs=False)
  File "/usr/lib64/python2.5/doctest.py", line 1351, in run
    self.debugger = _OutputRedirectingPdb(save_stdout)
  File "/usr/lib64/python2.5/doctest.py", line 324, in __init__
    pdb.Pdb.__init__(self, stdout=out)
  File "/usr/lib64/python2.5/pdb.py", line 57, in __init__
    cmd.Cmd.__init__(self, completekey, stdin, stdout)
  File "/usr/lib64/python2.5/cmd.py", line 90, in __init__
    import sys
  File "<doctest shorturl.txt[10]>", line 4, in fakeimport
NameError: global name 'realimport' is not defined

However, it does work when I run the same code from the python interactive console.

MORE EDIT:

I'm using zope.testing and a test file, shorturl.txt that has all the tests specific to this part of my module. First I'm importing the module with zope.component available, to demonstrate & test the usual usage. The absence of zope.* packages is considered an edge-case, so I'm testing it later. Thus, I have to reload() my module, after making zope.* unavailable, somehow.

So far I've even tried using tempfile.mktempdir() and empty zope/__init__.py and zope/component/__init__.py files in the tempdir, then inserting tempdir to sys.path[0], and removing the old zope.* packages from sys.modules.

Didn't work either.

EVEN MORE EDIT:

In the meantime, I've tried this:

>>> class NoZope(object):
...     def find_module(self, fullname, path):
...         if fullname.startswith('zope'):
...             raise ImportError
... 

>>> import sys
>>> sys.path.insert(0, NoZope())

And it works well for the namespace of the test suite (= for all imports in shorturl.txt), but it is not executed in my main module, ao.shorturl. Not even when I reload() it. Any idea why?

>>> import zope  # ok, this raises an ImportError
>>> reload(ao.shorturl)    <module ...>

Importing zope.interfaces raises an ImportError, so it doesn't get to the part where I import zope.component, and it remains in the ao.shorturl namespace. Why?!

>>> ao.shorturl.zope.component  # why?! 
<module ...>

Answer 1

Just monkeypatch into the builtins your own version of __import__ -- it can raise whatever you wish when it recognizes it's being called on the specific modules for which you want to mock up errors. See the docs for copious detail. Roughly:

try:
    import builtins
except ImportError:
    import __builtin__ as builtins
realimport = builtins.__import__

def myimport(name, globals, locals, fromlist, level):
    if ...:
        raise ImportError
    return realimport(name, globals, locals, fromlist, level)

builtins.__import__ = myimport

In lieu of the ..., you can hardcode name == 'zope.component', or arrange things more flexibly with a callback of your own that can make imports raise on demand in different cases, depending on your specific testing needs, without requiring you to code multiple __import__-alike functions;-).

Note also that if what you use, instead of import zope.component or from zope.component import something, is from zope import component, the name will then be 'zope', and 'component' will then be the only item in the fromlist.

Edit: the docs for the __import__ function say that the name to import is builtin (like in Python 3), but in fact you need __builtins__ -- I've edited the code above so that it works either way.

Answer 2

If you don't mind changing your program itself, you could also put the import call in a function and patch that in your tests.

Answer 3

This is what I justed in my unittests.

It uses PEP-302 "New Import Hooks". (Warning: the PEP-302 document and the more concise release notes I linked aren't exactly accurate.)

I use meta_path because it's as early as possible in the import sequence.

If the module has already been imported (as in my case, because earlier unittests mock against it), then it's necessary to remove it from sys.modules before doing the reload on the dependent module.

Ensure we fallback to using ~/.pif if XDG doesn't exist.

 >>> import sys

 >>> class _():
 ... def __init__(self, modules):
 ...  self.modules = modules
 ...
 ...  def find_module(self, fullname, path=None):
 ...  if fullname in self.modules:
 ...   raise ImportError('Debug import failure for %s' % fullname)

 >>> fail_loader = _(['xdg.BaseDirectory'])
 >>> sys.meta_path.append(fail_loader)

 >>> del sys.modules['xdg.BaseDirectory']

 >>> reload(pif.index) #doctest: +ELLIPSIS
 <module 'pif.index' from '...'>

 >>> pif.index.CONFIG_DIR == os.path.expanduser('~/.pif')
 True

 >>> sys.meta_path.remove(fail_loader)

Where the code inside pif.index looks like:

try:
    import xdg.BaseDirectory

    CONFIG_DIR = os.path.join(xdg.BaseDirectory.xdg_data_home, 'pif')
except ImportError:
    CONFIG_DIR = os.path.expanduser('~/.pif')

---

To answer the question about why the newly reloaded module has properties of the old and new loads, here are two example files.

The first is a module y with an import failure case.

# y.py

try:
    import sys

    _loaded_with = 'sys'
except ImportError:
    import os

    _loaded_with = 'os'

The second is x which demonstrates how leaving handles about for a module can affect its properties when being reloaded.

# x.py

import sys

import y

assert y._loaded_with == 'sys'
assert y.sys

class _():
    def __init__(self, modules):
        self.modules = modules

    def find_module(self, fullname, path=None):
        if fullname in self.modules:
            raise ImportError('Debug import failure for %s' % fullname)

# Importing sys will not raise an ImportError.
fail_loader = _(['sys'])
sys.meta_path.append(fail_loader)

# Demonstrate that reloading doesn't work if the module is already in the
# cache.

reload(y)

assert y._loaded_with == 'sys'
assert y.sys

# Now we remove sys from the modules cache, and try again.
del sys.modules['sys']

reload(y)

assert y._loaded_with == 'os'
assert y.sys
assert y.os

# Now we remove the handles to the old y so it can get garbage-collected.
del sys.modules['y']
del y

import y

assert y._loaded_with == 'os'
try:
    assert y.sys
except AttributeError:
    pass
assert y.os