How to capture all requests made by page in webdri

Question

I am using Selenium2/WebDriver to test my web applications. All the tests are written in Java and run with Maven.

While opening a page with webdriver I'd like to capture all the requests made by page (images, js and css files, etc). I use this data mainly for two reasons

• checking for 404 (and other errors) in calls
• checking if analytics code is working (checking if it's sending proper requests)

Depending on the project I use either Firebug with Netexport or Browsermob proxy. In both cases I can easily obtain a HAR (Html ARchive) file, parse it and extract the data I want.

Here's the problem: I am not happy with neither of these solutions. I have especially problems with getting HAR file when a page contains video that is being loaded too long. I am looking for something more stable.

So, the questions are:

Is there any alternative to Browsermob? I know about FiddlerCore but it's a .NET library and my tests are written in Java. I've also heard about Ajax DynaTrace and I know that there is some way to integrate it with Selenium but the documentation I found was for Selenium-RC not WebDriver.

Is there any way to integrate DynaTrace with WebDriver or use FiddlerCore with Java?

Is there any other way to achieve the goals I mentioned? I am looking for a proxy that I can easily control from my code. Exporting data to HAR would be a great plus.

Answer 1

There is an alternative with firefox ver 42+, there is addon called Firefox HarExport

File harExportApi = new File(System.getProperty("user.dir")
                     + "/src/main/resources/firebug/harexporttrigger-0.5.0-beta.7.xpi");

netExportProfile.addExtension(harExportApi);
netExportProfile.setPreference("extensions.netmonitor.har.enableAutomation", true);
    netExportProfile.setPreference("extensions.netmonitor.har.contentAPIToken", "test");
    netExportProfile.setPreference("extensions.netmonitor.har.autoConnect", true);

cap.setCapability(FirefoxDriver.PROFILE, netExportProfile);

and running following script will give us all the request responses

 String getHarLogScript = "var options = {\n" +
                "    token: \"test\",\n" +
                "    getData: true,\n" +
                "    title: \"my custom title\",\n" +
                "    jsonp: false,\n" +
                "  };\n" +
                "\n" +
                "  HAR.triggerExport(options).then(result => {\n" +
                "    var har = JSON.parse(result.data);\n" +
                "\n" +
                "    // Use performance.timing to provide onContentLoad\n" +
                "    +
                "     +
                "    var t = performance.timing;\n" +
                "    var pageTimings = har.log.pages[0].pageTimings;\n" +
                "    pageTimings.onContentLoad = t.domContentLoadedEventStart - t.navigationStart;\n" +
                "    pageTimings.onLoad = t.loadEventStart - t.navigationStart;\n" +
                "\n" +
                "    window.HarEntries=har.log.entries\n" +
                "\n" +
                "    console.log(\"HAR log (\" + result.data.length + \") \", har.log);\n" +
                "  }, err => {\n" +
                "    console.error(err);\n" +
                "  });"

LOG.info("Loading HAR log entries object into browser HarEntries object");
SeleniumUtils.executeScript(driver, getHarLogScript);

harEntries = ((List<Object>) SeleniumUtils.executeScript(driver, "return window.HarEntries"));

Answer 2

I found a google groups discussion on the topic. These links look like promising alternative to Browsermob:

• A Selenium CaptureNetworkTraffic Example in Java
• HOWTO: Collect WebDriver HTTP Request and Response Headers
• Automating the Capture of Web Timings with Selenium 2