How to get HTTP Response Code using Selenium WebDr

Question

I have written tests with Selenium2/WebDriver and want to test if HTTP Request returns an HTTP 403 Forbidden.

Is it possible to get the HTTP response status code with Selenium WebDriver?

Answer 1

You could try Mobilenium (https://github.com/rafpyprog/Mobilenium), a python package that binds BrowserMob Proxy and Selenium.

An usage example:

>>> from mobilenium import mobidriver
>>>
>>> browsermob_path = 'path/to/browsermob-proxy'
>>> mob = mobidriver.Firefox(browsermob_binary=browsermob_path)
>>> mob.get('http://python-requests.org')
301
>>> mob.response['redirectURL']
'http://docs.python-requests.org'
>>> mob.headers['Content-Type']
'application/json; charset=utf8'
>>> mob.title
'Requests: HTTP for Humans \u2014 Requests 2.13.0 documentation'
>>> mob.find_elements_by_tag_name('strong')[1].text
'Behold, the power of Requests'

Answer 2

For those people using Python, you might consider Selenium Wire, a library I recently developed for inspecting requests on a work project.

The library is designed to be lightweight, simple to use with minimal external dependencies and configuration. It provides your tests with a Pythonic API for accessing headers, status code, body content, as well as the ability to modify headers and rewrite URLs.

from seleniumwire import webdriver  # Import from seleniumwire

# Create a new instance of the Firefox driver
driver = webdriver.Firefox()

# Go to the Google home page
driver.get('https://www.google.com')

# Access requests via the `requests` attribute
for request in driver.requests:
    if request.response:
        print(
            request.path,
            request.response.status_code,
            request.response.headers['Content-Type']
        )

Prints:

https://www.google.com/ 200 text/html; charset=UTF-8
https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_120x44dp.png 200 image/png
https://consent.google.com/status?continue=https://www.google.com&pc=s&timestamp=1531511954&gl=GB 204 text/html; charset=utf-8
https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png 200 image/png
https://ssl.gstatic.com/gb/images/i2_2ec824b0.png 200 image/png
https://www.google.com/gen_204?s=webaft&t=aft&atyp=csi&ei=kgRJW7DBONKTlwTK77wQ&rt=wsrt.366,aft.58,prt.58 204 text/html; charset=UTF-8
...

Answer 3

In a word, no. It's not possible using the Selenium WebDriver API. This has been discussed ad nauseum in the issue tracker for the project, and the feature will not be added to the API.

Answer 4

You can use BrowserMob proxy to capture the requests and responses with a HttpRequestInterceptor. Here is an example in Java:

// Start the BrowserMob proxy
ProxyServer server = new ProxyServer(9978);
server.start();

server.addResponseInterceptor(new HttpResponseInterceptor()
{
    @Override
    public void process(HttpResponse response, HttpContext context)
        throws HttpException, IOException
    {
        System.out.println(response.getStatusLine());
    }
});

// Get selenium proxy
Proxy proxy = server.seleniumProxy();

// Configure desired capability for using proxy server with WebDriver
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, proxy);

// Set up driver
WebDriver driver = new FirefoxDriver(capabilities);

driver.get("http://stackoverflow.com/questions/6509628/webdriver-get-http-response-code");

// Close the browser
driver.quit();

Answer 5

It is not possible to get HTTP Response code by using Selenium WebDriver directly. The code can be got by using Java code and that can be used in Selenium WebDriver.

To get HTTP Response code by java:

public static int getResponseCode(String urlString) throws MalformedURLException, IOException{
    URL url = new URL(urlString);
    HttpURLConnection huc = (HttpURLConnection)url.openConnection();
    huc.setRequestMethod("GET");
    huc.connect();
    return huc.getResponseCode();
}

Now you can write your Selenium WebDriver code as below:

private static int statusCode;
public static void main(String... args) throws IOException{
    WebDriver driver = new FirefoxDriver();
    driver.manage().window().maximize();
    driver.get("https://www.google.com/");
    driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);

    List<WebElement> links = driver.findElements(By.tagName("a"));
    for(int i = 0; i < links.size(); i++){
        if(!(links.get(i).getAttribute("href") == null) && !(links.get(i).getAttribute("href").equals(""))){
            if(links.get(i).getAttribute("href").contains("http")){
                statusCode= getResponseCode(links.get(i).getAttribute("href").trim());
                if(statusCode == 403){
                    System.out.println("HTTP 403 Forbidden # " + i + " " + links.get(i).getAttribute("href"));
                }
            }
        }   
    }   
}

Answer 6

I was also having same issue and stuck for some days, but after some research i figured out that we can actually use chrome's "--remote-debugging-port" to intercept requests in conjunction with selenium web driver. Use following Pseudocode as a reference:-

create instance of chrome driver with remote debugging

int port = findFreePort();

chromeOptions.addArguments("--remote-debugging-port=" + freePort);

ChromeDriver driver = new ChromeDriver(chromeOptions);`

make a get call to http://127.0.0.1:freePort

String response = makeGetCall( "http://127.0.0.1" + freePort  + "/json" );

Extract chrome's webSocket Url to listen, you can see response and figure out how to extract

String webSocketUrl = response.substring(response.indexOf("ws://127.0.0.1"), response.length() - 4);

Connect to this socket, u can use asyncHttp

socket = maketSocketConnection();

Enable network capture

socket.send( { "id" : 1, "method" : "Network.enable" } );

Now chrome will send all network related events and captures them as follows

socket.onMessageReceived( String message ){

    Json responseJson = toJson(message);
    if( responseJson.method == "Network.responseReceived" ){
       //extract status code
    }
}

driver.get("http://stackoverflow.com");

you can do everything mentioned in dev tools site. see https://chromedevtools.github.io/devtools-protocol/ Note:- use chromedriver 2.39 or above.

I hope it helps someone.

reference : Using Google Chrome remote debugging protocol