How to access the values of Chrome's Dev tools

Question

I'm using chrome option to access the performance logging using selenium, I'm trying to write a code that would help me figure out the total number of the http request and the size of the page after the loading is finished. Manually we can check this by using Dev tool's network tab. Just need to know how to access the Network table's value or summary values. Because the performance logging is not giving me the summarized values that I need, I would like to write a code to get:

The total number of requests=

What the total weight of the Page is=

If it is possible.

Screen shot of network tab highlighted the Summary and Request table values that i need to access

capabilities = DesiredCapabilities.CHROME
capabilities['loggingPrefs'] = {'browser': 'DEBUG'}
capabilities['loggingPrefs'] = {'performance': 'ALL'}
capabilities['perfLoggingPrefs'] = {'enableTimeline': 'true'}


driverLocation = "/Users/harisrizwan/Selenium/chrome/chromedriver"
os.environ["chrome.driver"] = driverLocation
chrome_options = Options()
chrome_options.add_argument("headless")

driver= 
webdriver.Chrome(driverLocation,desired_capabilities=capabilities)
driver.implicitly_wait(10)
driver.maximize_window()
baseUrl="www.google.com"
driver.get(baseUrl)

using pandas to create a dataframe of the log.

df = pd.DataFrame(driver.get_log('performance'))
df.to_clipboard(index=False)

Thank you.

Answer 1

I"m not sure on calculating total weight of the page but obtaining total number of request sent to the server is possible

Use https://github.com/lightbody/browsermob-proxy and add the proxy to the desired_capabilities, once the script finished, dump the har file to json and get all the request

in python:

from browsermobproxy import Server
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import json

server = Server('path to the proxy server file')
server.start()
proxy = server.create_proxy()

options = Options()
options.add_argument(f'--proxy-server={proxy.proxy}')
driver = webdriver.Chrome('path to chromedriver', desired_capabilities=options.to_capabilities())

proxy.new_har()
driver.get('https://www.google.com')

result = json.dumps(proxy.har)
json_data = json.loads(result)

request= [x for x in json_data['log']['entries']]
server.stop()
driver.close()

Answer 2

You could use the performance API to get the transferred size.

Transferred size for the main page and each resources:

sizes = driver.execute_script("""
  return performance.getEntries()
    .filter(e => e.entryType==='navigation' || e.entryType==='resource')
    .map(e=> ([e.name, e.transferSize]));
  """)

Transferred size for the main page only:

size = driver.execute_script("""
  return performance.getEntriesByType('navigation')[0].transferSize;
  """)

Total transferred size for the main page and resources:

size = driver.execute_script("""
  return performance.getEntries()
    .filter(e => e.entryType==='navigation' || e.entryType==='resource')
    .reduce((acc, e) => acc + e.transferSize, 0)
  """)