How to Mock an HTTP request in a unit testing scen

I would like to include a Web server for all my test related to HTTP. It doesn't need to be very sophisticated. I would prefer not to be dependent of being online. So I could test some options of my program.

Start the server
Create a few resources (URI) with appropriate mime types, response code, etc.
Run the tests (would be good to not have to start the server for each tests too)
Shut down the server.

Any hints on this code would be helpful. I tried a few things with BaseHTTPServer but not successful yet. nosetests command seems to wait indefinitely.

import unittest
from foo import core

class HttpRequests(unittest.TestCase):
    """Tests for HTTP"""

    def setUp(self):
        "Starting a Web server"
        self.port = 8080
        # Here we need to start the server
        #
        # Then define a couple of URIs and their HTTP headers
        # so we can test the code.
        pass

    def testRequestStyle(self):
        "Check if we receive a text/css content-type"
        myreq = core.httpCheck()
        myuri = 'http://127.0.0.1/style/foo'
        myua = "Foobar/1.1"
        self.asserEqual(myreq.mimetype(myuri, myua), "text/css")

    def testRequestLocation(self):
        "another test" 
        pass

    def tearDown(self):
        "Shutting down the Web server"
        # here we need to shut down the server
        pass

thanks for any help.

Update - 2012:07:10T02:34:00Z

This is a code which for a given Web site will return the list of CSS. I want to test if it returns the right list of CSS.

import unittest
from foo import core

class CssTests(unittest.TestCase):
    """Tests for CSS requests"""

    def setUp(self):
        self.css = core.Css()
        self.req = core.HttpRequests()

    def testCssList(self):
        "For a given Web site, check if we get the right list of linked stylesheets"
        WebSiteUri = 'http://www.opera.com/'
        cssUriList = [
        'http://www.opera.com/css/handheld.css',
        'http://www.opera.com/css/screen.css',
        'http://www.opera.com/css/print.css',
        'http://www.opera.com/css/pages/home.css']
        content = self.req.getContent(WebSiteUri)
        cssUriListReq = self.css.getCssUriList(content, WebSiteUri)
        # we need to compare ordered list.
        cssUriListReq.sort()
        cssUriList.sort()
        self.assertListEqual(cssUriListReq, cssUriList)

Then in foo/core.py

import urlparse
import requests
from lxml import etree
import cssutils

class Css:
    """Grabing All CSS for one given URI"""


    def getCssUriList(self, htmltext, uri):
        """Given an htmltext, get the list of linked CSS"""
        tree = etree.HTML(htmltext)
        sheets = tree.xpath('//link[@rel="stylesheet"]/@href')
        for i, sheet in enumerate(sheets):
            cssurl = urlparse.urljoin(uri, sheet)
            sheets[i] = cssurl
        return sheets

Right now, the code depends on an online server. It should not. I want to be able to add plenty of different types of combination of stylesheets and to test the protocol and then later on some options on their parsing, combinations, etc.

Starting a web server for unit testing is definitely not a good practice. Unit tests should be simple and isolated, which means that they should avoid performing IO operations for example.

If what you want to write are really unit tests then you should craft your own test inputs and also look into mock objects. Python being a dynamic language, mocking and monkey pathing are easy and powerful tools for writing unit test. In particular, have a look at the excellent Mock module.

Simple unit test

So, if we have a look at your CssTests example, you are trying to test that css.getCssUriList is able to extract all the CSS stylesheet referenced in a piece of HTML you give it. What you are doing in this particular unit test is not testing that you can send a request and get a response from a website, right? You simply want to make sure that given some HTML, your function returns the correct list of CSS URLs. So, in this test, you clearly do not need to talk to a real HTTP server.

I would do something like the following:

import unittest

class CssListTestCase(unittest.TestCase):

    def setUp(self):
        self.css = core.Css()

    def test_css_list_should_return_css_url_list_from_html(self):
        # Setup your test
        sample_html = """
        <html>
            <head>
                <title>Some web page</title>
                <link rel='stylesheet' type='text/css' media='screen'
                      href='http://example.com/styles/full_url_style.css' />
                <link rel='stylesheet' type='text/css' media='screen'
                      href='/styles/relative_url_style.css' />
            </head>
            <body><div>This is a div</div></body>
        </html>
        """
        base_url = "http://example.com/"

        # Exercise your System Under Test (SUT)
        css_urls = self.css.get_css_uri_list(sample_html, base_url)

        # Verify the output
        expected_urls = [
            "http://example.com/styles/full_url_style.css",
            "http://example.com/styles/relative_url_style.css"
        ]
        self.assertListEqual(expected_urls, css_urls)

Mocking with Dependency Injection

Now, something less obvious would be unit testing the getContent() method of your core.HttpRequests class. I suppose you are using an HTTP library and not making your own requests on top of TCP sockets.

To keep your tests at the unit level, you don't want to send anything over the wire. What you can do to avoid that, is having tests that ensure that you make use of your HTTP library correctly. This is about testing not the behaviour of your code but rather the way it interacts with the other objects around it.

One way to do so would be to make the dependency on that library explicit: we can add a parameter to the HttpRequests.__init__ to pass it an instance of library's HTTP client. Say I use an HTTP library that provides a HttpClient object on which we can call get(). You could do something like:

class HttpRequests(object):

    def __init__(self, http_client):
        self.http_client = http_client

   def get_content(self, url):
        # You could imagine doing more complicated stuff here, like checking the
        # response code, or wrapping your library exceptions or whatever
        return self.http_client.get(url)

We have made the dependency explicit and the requirement now needs to be met by the caller of HttpRequests: this is called Dependency Injection (DI).

DI is very useful for two things:

it avoids surprises where your code is secretely relying on some object to exist somewhere
it allows test to be written that inject different sort of objects depending on the goal of that test

Here, we can use a mock object that we will give to core.HttpRequests and that it will use, unknowingly, as if it were the real library. After that, we can test that the interaction was conducted as expected.

import core

class HttpRequestsTestCase(unittest.TestCase):

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # We create an object that is not a real HttpClient but that will have
        # the same interface (see the `spec` argument). This mock object will
        # also have some nice methods and attributes to help us test how it was used.
        mock_http_client = Mock(spec=somehttplib.HttpClient) 

        # Exercise

        http_requests = core.HttpRequests(mock_http_client)
        content = http_requests.get_content(url)

        # Here, the `http_client` attribute of `http_requests` is the mock object we
        # have passed it, so the method that is called is `mock.get()`, and the call
        # stops in the mock framework, without a real HTTP request being sent.

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        mock_http_client.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = mock_http_client.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received it
        self.assertEqual(content, expected_content)

We have now tested that our get_content method interacts correctly with our HTTP library. We have defined the boundaries of our HttpRequests object and tested them, and this is as far as we should go at the unit test level. The request is now in the hand of that library and it is certainly not the role of our unit test suite to test that the library works as expected.

Monkey patching

Now imagine that we decide to use the great requests library. Its API being more procedural, it does not present an object we can grab to make HTTP requests from. Instead, we would import the module and call its get method.

Our HttpRequests class in core.py would then look somethings like the following:

import requests

class HttpRequests(object):

    # No more DI in __init__

    def get_content(self, url):
        # We simply delegate the HTTP work to the `requests` module
        return requests.get(url)

No more DI, so now, we are left wondering:

How am I to prevent network interaction from happening?
How am I to test that I use the requests module properly?

This is where you can use another fantastic, yet controversial, mechanism that dynamic languages offer: monkey patching. We will replace, at runtime, the requests module with an object we craft and can use in our test.

Our unit test will then look something like:

import core

class HttpRequestsTestCase(unittest.TestCase):

    def setUp(self):
        # We create a mock to replace the `requests` module
        self.mock_requests = Mock()

        # We keep a reference to the current, real, module
        self.old_requests = core.requests

        # We replace the module with our mock
        core.requests = self.mock_requests

    def tearDown(self):
        # It is very important that each unit test be isolated, so we need
        # to be good citizen and clean up after ourselves. This means that
        # we need to put back the correct `requests` module where it was
        core.requests = self.old_requests

    def test_get_content_should_use_get_properly(self):
        # Setup

        url = "http://example.com"

        # Exercise
        http_client = core.HttpRequests()
        content = http_client.get_content(url)

        # Verify

        # We expect our get_content method to have called our http library.
        # Let's check!
        self.mock_requests.get.assert_called_with(url)

        # We can find out what our mock object has returned when get() was
        # called on it
        expected_content = self.mock_requests.get.return_value
        # Since our get_content returns the same result without modification,
        # we should have received
        self.assertEqual(content, expected_content)

To make this process less verbose, the mock module has a patch decorator that looks after the scaffolding. We then only need to write:

import core

class HttpRequestsTestCase(unittest.TestCase):

    @patch("core.requests")
    def test_get_content_should_use_get_properly(self, mock_requests):
        # Notice the extra param in the test. This is the instance of `Mock` that the
        # decorator has substituted for us and it is populated automatically.

        ...

        # The param is now the object we need to make our assertions against
        expected_content = mock_requests.get.return_value

Conclusion

It is very important to keep unit test small, simple, fast, and self-contained. A unit test that relies on another server to be running is simply not a unit test. To help with that, DI is a great practice, and mock objects a great tool.

At first, it is not easy to get your head around the concept of mock and how to use them though. Like every power tool, they can also explode in your hands and for example make you believe you have tested something when in reality you have not. Making sure that the behaviour and input/output of mock objects reflects the reality is paramount.

P.S.

Given that we have never interacted with a real HTTP server at the unit test level, it is important to write Integration Tests that will make sure our application is able to talk to the sort of servers it will deal with in real life. We could do this with a fully fledged server set up specially for Integration Testing, or write a contrived one.