How to read nested JSON structure in R?

Question

I have some JSON that looks like this:

"total_rows":141,"offset":0,"rows":[
{"id":"1","key":"a","value":{"SP$Sale_Price":"240000","CONTRACTDATE$Contract_Date":"2006-10-26T05:00:00"}},
{"id":"2","key":"b","value":{"SP$Sale_Price":"2000000","CONTRACTDATE$Contract_Date":"2006-08-22T05:00:00"}},
{"id":"3","key":"c","value":{"SP$Sale_Price":"780000","CONTRACTDATE$Contract_Date":"2007-01-18T06:00:00"}},
...

In R, what would be the easiest way to produce a scatter-plot of SP$Sale_Price versus CONTRACTDATE$Contract_Date?

I got this far:

install.packages("rjson")
library("rjson")
json_file <- "http://localhost:5984/testdb/_design/sold/_view/sold?limit=100"
json_data <- fromJSON(file=json_file)
install.packages("plyr")
library(plyr)
asFrame <- do.call("rbind.fill", lapply(json_data, as.data.frame))

but now I'm stuck...

> plot(CONTRACTDATE$Contract_Date, SP$Sale_Price)
Error in plot(CONTRACTDATE$Contract_Date, SP$Sale_Price) : 
  object 'CONTRACTDATE' not found

How to make this work?

Answer 1

I found a way that doesn't discard the field names:

install.packages("jsonlite")
install.packages("curl")
json <- fromJSON(json_file)
r <- json$rows

At this point r looks like this:

> class(r)
[1] "data.frame"
> colnames(r)
[1] "id"    "key"   "value"

After some more Googling and trial-and-error I landed on this:

f <- r$value
sp <- strtoi(f[["SP$Sale_Price"]])
cd <- strptime(f[["CONTRACTDATE$Contract_Date"]], format="%Y-%m-%dT%H:%M:%S", tz="GMT")
plot(cd,sp)

And the result on my full data-set...

Answer 2

Suppose you have the following JSON-file:

txt <- '{"total_rows":141,"offset":0,"rows":[
  {"id":"1","key":"a","value":{"SP$Sale_Price":"240000","CONTRACTDATE$Contract_Date":"2006-10-26T05:00:00"}},
  {"id":"2","key":"b","value":{"SP$Sale_Price":"2000000","CONTRACTDATE$Contract_Date":"2006-08-22T05:00:00"}},
  {"id":"3","key":"c","value":{"SP$Sale_Price":"780000","CONTRACTDATE$Contract_Date":"2007-01-18T06:00:00"}}]}'

Then you can read it as follows with the jsonlite package:

library(jsonlite)
json_data <- fromJSON(txt, flatten = TRUE)

# get the needed dataframe
dat <- json_data$rows
# set convenient names for the columns
# this step is optional, it just gives you nicer columnnames
names(dat) <- c("id","key","sale_price","contract_date")
# convert the 'contract_date' column to a datetime format
dat$contract_date <- strptime(dat$contract_date, format="%Y-%m-%dT%H:%M:%S", tz="GMT")

Now you can plot:

plot(dat$contract_date, dat$sale_price)

Which gives:

---

If you choose not to flatten the JSON, you can do:

json_data <- fromJSON(txt)

dat <- json_data$rows$value

sp <- strtoi(dat$`SP$Sale_Price`)
cd <- strptime(dat$`CONTRACTDATE$Contract_Date`, format="%Y-%m-%dT%H:%M:%S", tz="GMT")
plot(cd,sp)

Which gives the same plot: