Extract from dynamic JSON response with Scrapy

Question

I want to extract the 'avail' value from the JSON output that look like this.

{
    "result": {
        "code": 100,
        "message": "Command Successful"
    },
    "domains": {
        "yolotaxpayers.com": {
            "avail": false,
            "tld": "com",
            "price": "49.95",
            "premium": false,
            "backorder": true
        }
    }
}

The problem is that the ['avail'] value is under ["domains"]["domain_name"] and I can't figure out how to get the domain name.

You have my spider below. The first part works fine, but not the second one.

import scrapy
import json
from whois.items import WhoisItem

class whoislistSpider(scrapy.Spider):
    name = "whois_list"
    start_urls = []
    f = open('test.txt', 'r')
    global lines
    lines = f.read().splitlines()
    f.close()
    def __init__(self):
        for line in lines:
            self.start_urls.append('http://www.example.com/api/domain/check/%s/com' % line)

    def parse(self, response):
        for line in lines:
            jsonresponse = json.loads(response.body_as_unicode())
            item = WhoisItem()
            domain_name = list(jsonresponse['domains'].keys())[0]
            item["avail"] = jsonresponse["domains"][domain_name]["avail"]
            item["domain"] = domain_name
            yield item

Thank you in advance for your replies.

Answer 1

To get the domain name from above json response you can use list comprehension , e.g:

domain_name = [x for x in jsonresponse.values()[0].keys()]

To get the "avail" value use same method, e.g:

avail = [x["avail"] for x in jsonresponse.values()[0].values() if "avail" in x]

to get the values in string format you should call it by index 0 e.g:

domain_name[0] and avail[0] because list comprehension results stored in list type variable.

More info on list comprehension

Answer 2

Assuming you are only expecting one result per response:

domain_name = list(jsonresponse['domains'].keys())[0]
item["avail"] = jsonresponse["domains"][domain_name]["avail"]

This will work even if there is a mismatch between the domain in the file "test.txt" and the domain in the result.

Answer 3

Currently, it tries to get the value by the "('%s.com' % line)" key.

You need to do the string formatting correctly:

domain_name = "%s.com" % line.strip()
item["avail"] = jsonresponse["domains"][domain_name]["avail"]