Script to Mobile-Friendly test

Question

I wanted to write a shell/python script which will check if a website is mobile friendly or not. Using browser this can be easily done by visiting-

https://www.google.com/webmasters/tools/mobile-friendly/?url=<website_addr>

For eg.-

https://www.google.com/webmasters/tools/mobile-friendly/?url=http://facebook.com

I tried fetching the content through curl, wget, lynx commands but it did not worked.

How can I do so?

Answer 1

The page uses a JSONP request to an as-yet unpublished Google PageSpeed API. Google publishes PageSpeeds Insights API v2, but the page appears to be using a v3beta1 endpoint.

When you go to the https://www.google.com/webmasters/tools/mobile-friendly/?url=http://facebook.com page for example and look at the network tab of your browser developer tools, you'll see a request for:

https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?key=AIzaSyDkEX-f1JNLQLC164SZaobALqFv4PHV-kA&screenshot=true&snapshots=true&locale=en_US&url=http%3A%2F%2Ffacebook.com%2F&strategy=mobile&filter_third_party_resources=false&callback=_callbacks_._Ce2bYp0wchLY

The url parameter is directly taken from the url parameter passed to the page, the callback parameter is there for the JSONP request to provide a callback wrapper.

There is a chance Google will swap out the API key used there, but in the meantime you can use Python code to validate the mobile friendliness of a site with:

import requests

url_to_test = 'http://facebook.com'

params = {
    'key': 'AIzaSyDkEX-f1JNLQLC164SZaobALqFv4PHV-kA',
    'url': url_to_test,
}
api_url = 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady'
response = requests.get(api_url, params=params)
data = response.json()
passed = all(rule['pass'] for rule in data['ruleGroups'].values())

print('{} is {}'.format(url_to_test, 'mobile friendly' if passed else 'not mobile friendly'))

Answer 2

Sanchit,

I suggest you look at the requests library for retrieving the url. Also, as has already been said (I don't have experience with this api) you need to call 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebook.com' instead of the url you posted.

Here's an example:

import requests

r = requests.get('https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebook.com')

data = r.json()

That would give you a json file with all the data that the website you posted uses.

Answer 3

Solved it myself, with help of @TimberlakeCoding & @MartijnPieters. Here it is-

$ wget -q -O - https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=http://facebo‌​ok.com | grep "\"pass\": true" 

If the exit status code is 0, that means website is mobile friendly else not.

Hope it helps someone! Thanks

Answer 4

I wrote a simple python script for this similar task to send multiple network requests to google Mobile-Friendly Test api and save "pass" and some other fields to mysql db. It's very fast and efficient.

# download mysql connector for python 
# from: https://dev.mysql.com/downloads/connector/odbc/
# select your Platform from drop-down and install it

    from twisted.internet import reactor, threads
    from urlparse import urlparse
    import httplib
    import itertools
    import json
    import mysql.connector

    GOOGLE_API_KEY = 'YOUR GOOGLE API KEY HERE'

    db = mysql.connector.connect(user='root', password='root',
                                  host='127.0.0.1',
                                  database='mobiletracker', autocommit=True)


    cursor = db.cursor()

    concurrent = 10
    finished=itertools.count(1)
    reactor.suggestThreadPoolSize(concurrent)

    def getData(ourl):
        googleapiUrl = 'https://www.googleapis.com/pagespeedonline/v3beta1/mobileReady?url=' + ourl + '&key=' + GOOGLE_API_KEY
        print googleapiUrl
        url = urlparse(googleapiUrl)
        conn = httplib.HTTPSConnection(url.netloc)   
        conn.request("GET", url.path + '?' + url.query)
        res = conn.getresponse()
        return res.read()

    def processResponse(response,url):
        jsonData = json.loads(response)
        try:
          score = str(jsonData['ruleGroups']['USABILITY']['score'])
        except Exception, e:
          score = '0'
        try:
          pass_ = jsonData['ruleGroups']['USABILITY']['pass']  #Boolean
          if pass_:
            pass_ = '1'
          else:
            pass_ = '0'
        except Exception, e:
          pass_ = '0'
        try:
          cms = str(jsonData['pageStats']['cms'])
        except Exception, e:
          cms = ''

        cursor.execute("SELECT id FROM mobile WHERE url='" + url + "'")
        result = cursor.fetchone()
        try:
          id_ = str(result[0])
          query = "UPDATE mobile SET score='" + score + "', pass='" + pass_ + "', cms='" + cms + "' WHERE id = '" + id_ + "'"
          print query
          cursor.execute(query)
        except Exception, e:
          query = "INSERT INTO mobile SET url='" + url + "', score='" + score + "', pass='" + pass_ + "', cms='" + cms + "'"
          print query
          cursor.execute(query)
        processedOne()

    def processError(error,url):
        print "error", url, error
        processedOne()

    def processedOne():
        if finished.next()==added:
            reactor.stop()

    def addTask(url):
        req = threads.deferToThread(getData, url)
        req.addCallback(processResponse, url)
        req.addErrback(processError, url)   

    added=0
    for url in open('urllist.csv'):
        added+=1
        addTask(url.strip())

    try:
        reactor.run()
    except KeyboardInterrupt:
        reactor.stop()

Also available at https://github.com/abm-adnan/multiple-requests