Python - make a POST request using Python 3 urllib

问题:

I am trying to make a POST request to the following page:http://search.cpsa.ca/PhysicianSearch

In order to simulate clicking the 'Search' button without filling out any of the form, which adds data to the page. I got the POST header information by clicking on the button while looking at the network tab in chrome developer tools. The reason I'm posting this instead of just copying solutions from the other similar problems is that I believe I may have not gotten the correct header information. Is it properly formatted and did I grab the right information? I've never made a POST request before.

This is what I've managed to piece together:

import urllib.parse
import urllib.request


data = urllib.parse.urlencode({'Host': 'search.cpsa.ca', 'Connection': 'keep-alive', 'Content-Length': 23796,
                                     'Origin': 'http://search.cpsa.ca', 'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
                                     'Cahce-Control': 'no-cache', 'X-Requested-With': 'XMLHttpRequest',
                                     'X-MicrosoftAjax': 'Delta=true', 'Accept': '*/*',
                                     'Referer': 'http://search.cpsa.ca/PhysicianSearch',
                                     'Accept-Encoding': 'gzip, deflate',
                                     'Accept-Language': 'en-GB,en-US;q=0.8,en;q=0.6',
                                     'Cookie': 'ASP.NET_SessionId=kcwsgio3dchqjmyjtwue402c; _ga=GA1.2.412607756.1459536682; _gat=1'})


url = "http://www.musi-cal.com/cgi-bin/query?%s"

data = data.encode('ascii')
with urllib.request.urlopen("http://search.cpsa.ca/PhysicianSearch", data) as f:
    print(f.read().decode('utf-8'))

This solution outputs the page's HTML, but not with any of the data I wanted to retrieve from the POST request.

回答1:

This is how you do it.

from urllib import request, parse
data = parse.urlencode(<your data dict>).encode()
req =  request.Request(<your url>, data=data) # this will make the method "POST"
resp = request.urlopen(req)

回答2:

Thank you C Panda. You really made it easy for me to learn this module.

I released the dictionary that we pass does not encode for me. I had to do a minor change -

from urllib import request, parse
import json

# Data dict
data = { 'test1': 10, 'test2': 20 }

# Dict to Json
# Difference is { "test":10, "test2":20 }
data = json.dumps(data)

# Convert to String
data = str(data)

# Convert string to byte
data = data.encode('utf-8')

# Post Method is invoked if data != None
req =  request.Request(<your url>, data=data)

# Response
resp = request.urlopen(req)

回答3:

The above code encoded the JSON string with some extra \" that caused me a lot of problems. This looks like a better way of doing it:

from urllib import request, parse

url = "http://www.example.com/page"

data = {'test1': 10, 'test2': 20}
data = parse.urlencode(data).encode()

req = request.Request(url, data=data)
response = request.urlopen(req)

print (response.read())