Listening for HTTP responses

问题:

I am trying to write a python application that will listen for HTTP responses on a socket. I am using http-parser for this. Here is my code:

#!/usr/bin/env python
import socket

from http_parser.http import HttpStream
from http_parser.reader import SocketReader

from http_parser.util import b

def main():
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((socket.gethostname(), 7000))
    s.listen(5)
    try:
        while True:
            p = HttpStream(SocketReader(s))
    finally:
        s.close()

if __name__ == "__main__":
    main()

I have two questions:

1. Is this the best way to do this? Note that I do not want to send a request and then listen for a response. I want to have a redirection mechanism that redirects all responses to the box running this script.
2. How should I test this? Is there a tool that can mock HTTP responses?

EDIT What I am trying to do is this: I have three boxes, once runs Apache, one runs this script and one is the client. When the client connects to Apache and it sends back a response, I am diverting the response to this box. So in this script, I am trying to listen for HTTP responses.

Topology Here is my topology: Server <----> Switch one <-----> Switch two <-----> Box one and two Initially, box one connects to the server and sends a request. When the second switch receives responses from the server, it forks it to both box one and box two.

回答1:

1. That is a perfectly fine implementation if you really want to operate at the level of TCP sockets. If you want more abstraction, there are lots and lots of HTTP server packages for python, including the standard library's BaseHttpServer, and external libraries/frameworks like tornado and cherrypy.

2. To test your HTTP listener there are lots of options. You could write full-on HTTP client test code in Python (using an HTTP client library like urllib), or you could:

   • Point your web browser to http://localhost:7000
   • telnet to port 7000 on localhost and type in raw HTTP requests.

   • Automate the above by using nc, e.g.:

     echo -e 'GET / HTTP/1.1\n\n' | nc localhost 7000
     

A note on terminology: What you are listening for on your bound socket is an HTTP request, in the HTTP parlance; what you send back to the connecting client is a response.

回答2:

1. Is there is a better way?

Yes, there is

2. How should I test this?

common practice is to put a test.py in the same folder, and run python test.py to test. Sample code:

#!python
# -*- coding: utf-8 -*-
import optparse
import urllib2
from django.utils import unittest

# just a sample of settings
HOST = 'localhost'

class GIOPTest(unittest.TestCase):
    def test_basic(self):
        #TEST GOES HERE

if __name__ == '__main__':
    # This part is to accept command line parameters
    option_list = ( # check optparse help for more options
        make_option("--host",
                action = 'store',
                type    = 'string',
                dest    = 'host',
                default = 'localhost',
                help    = 'Server host (localhost by default)'
            ),
    )

    parser = OptionParser(option_list=option_list)
    options, args = parser.parse_args()
    HOST = options.host

    # run the test
    unittest.main()

回答3:

I ended up using a raw socket and directly reading packets from it.

回答4:

if you're using apache as the server you're redirecting the data to you could use apache benchmark (ab - you must be superuser though to use it as far as I know) to test it... will also help evaluate the performance impact your application has on the whole thing, example

 ab -n 500 -c 20 http://localhost:7000/

where after -n is the total number of connections made to the server during the test and after -c you have the number of concurrent connections apache benchmark will make, experiment with different values of these, also read it's manual, you might find more useful options for whatever your specific purpose might be