Play [Scala]: How to flatten a JSON object

Question

Given the following JSON...

{
  "metadata": {
    "id": "1234",
    "type": "file",
    "length": 395
  }
}

... how do I convert it to

{
  "metadata.id": "1234",
  "metadata.type": "file",
  "metadata.length": 395
}

Tx.

Answer 1

This is definitely not trivial, but possible by trying to flatten it recursively. I haven't tested this thoroughly, but it works with your example and some other basic one's I've come up with using arrays:

object JsFlattener {

    def apply(js: JsValue): JsValue = flatten(js).foldLeft(JsObject(Nil))(_++_.as[JsObject])

    def flatten(js: JsValue, prefix: String = ""): Seq[JsValue] = {
        js.as[JsObject].fieldSet.toSeq.flatMap{ case (key, values) =>
            values match {
                case JsBoolean(x) => Seq(Json.obj(concat(prefix, key) -> x))
                case JsNumber(x) => Seq(Json.obj(concat(prefix, key) -> x))
                case JsString(x) => Seq(Json.obj(concat(prefix, key) -> x))
                case JsArray(seq) => seq.zipWithIndex.flatMap{ case (x, i) => flatten(x, concat(prefix, key + s"[$i]")) }  
                case x: JsObject => flatten(x, concat(prefix, key))
                case _ => Seq(Json.obj(concat(prefix, key) -> JsNull))
            }
        }
    }

    def concat(prefix: String, key: String): String = if(prefix.nonEmpty) s"$prefix.$key" else key

}

JsObject has the fieldSet method that returns a Set[(String, JsValue)], which I mapped, matched against the JsValue subclass, and continued consuming recursively from there.

You can use this example by passing a JsValue to apply:

val json = Json.parse("""
    {
      "metadata": {
        "id": "1234",
        "type": "file",
        "length": 395
      }
    }
"""
JsFlattener(json)

We'll leave it as an exercise to the reader to make the code more beautiful looking.

Answer 2

Here's my take on this problem, based on @Travis Brown's 2nd solution.

It recursively traverses the json and prefixes each key with its parent's key.

def flatten(js: JsValue, prefix: String = ""): JsObject = js.as[JsObject].fields.foldLeft(Json.obj()) {
    case (acc, (k, v: JsObject)) => {
        if(prefix.isEmpty) acc.deepMerge(flatten(v, k))
        else acc.deepMerge(flatten(v, s"$prefix.$k"))
    }
    case (acc, (k, v)) => {
        if(prefix.isEmpty) acc + (k -> v)
        else acc + (s"$prefix.$k" -> v)
    }
}

which turns this:

{
  "metadata": {
    "id": "1234",
    "type": "file",
    "length": 395
  },
  "foo": "bar",
  "person": {
    "first": "peter",
    "last": "smith",
    "address": {
      "city": "Ottawa",
      "country": "Canada"
    }
  }
}

into this:

{
  "metadata.id": "1234",
  "metadata.type": "file",
  "metadata.length": 395,
  "foo": "bar",
  "person.first": "peter",
  "person.last": "smith",
  "person.address.city": "Ottawa",
  "person.address.country": "Canada"
}

Answer 3

@Trev has the best solution here, completely generic and recursive, but it's missing a case for array support. I'd like something that works in this scenario:

turn this:

{
  "metadata": {
    "id": "1234",
    "type": "file",
    "length": 395
  },
  "foo": "bar",
  "person": {
    "first": "peter",
    "last": "smith",
    "address": {
      "city": "Ottawa",
      "country": "Canada"
    },
    "kids": ["Bob", "Sam"]
  }
}

into this:

{
  "metadata.id": "1234",
  "metadata.type": "file",
  "metadata.length": 395,
  "foo": "bar",
  "person.first": "peter",
  "person.last": "smith",
  "person.address.city": "Ottawa",
  "person.address.country": "Canada",
  "person.kids[0]": "Bob",
  "person.kids[1]": "Sam"
}

I've arrived at this, which appears to work, but seems overly verbose. Any help in making this pretty would be appreciated.

def flatten(js: JsValue, prefix: String = ""): JsObject = js.as[JsObject].fields.foldLeft(Json.obj()) {
  case (acc, (k, v: JsObject)) => {
    val nk = if(prefix.isEmpty) k else s"$prefix.$k"
    acc.deepMerge(flatten(v, nk))
  }
  case (acc, (k, v: JsArray)) => {
    val nk = if(prefix.isEmpty) k else s"$prefix.$k"
    val arr = flattenArray(v, nk).foldLeft(Json.obj())(_++_)
    acc.deepMerge(arr)
  }
  case (acc, (k, v)) => {
    val nk = if(prefix.isEmpty) k else s"$prefix.$k"
    acc + (nk -> v)
  }
}

def flattenArray(a: JsArray, k: String = ""): Seq[JsObject] = {
  flattenSeq(a.value.zipWithIndex.map {
    case (o: JsObject, i: Int) =>
      flatten(o, s"$k[$i]")
    case (o: JsArray, i: Int) =>
      flattenArray(o, s"$k[$i]")
    case a =>
      Json.obj(s"$k[${a._2}]" -> a._1)
  })
}

def flattenSeq(s: Seq[Any], b: Seq[JsObject] = Seq()): Seq[JsObject] = {
  s.foldLeft[Seq[JsObject]](b){
    case (acc, v: JsObject) =>
      acc:+v
    case (acc, v: Seq[Any]) =>
      flattenSeq(v, acc)
  }
}

Answer 4

You can do this pretty concisely with Play's JSON transformers. The following is off the top of my head, and I'm sure it could be greatly improved on:

import play.api.libs.json._

val flattenMeta = (__ \ 'metadata).read[JsObject].flatMap(
  _.fields.foldLeft((__ \ 'metadata).json.prune) {
    case (acc, (k, v)) => acc andThen __.json.update(
      Reads.of[JsObject].map(_ + (s"metadata.$k" -> v))
    )
  }
)

And then:

val json = Json.parse("""
  {
    "metadata": {
      "id": "1234",
      "type": "file",
      "length": 395
    }
  }
""")

And:

scala> json.transform(flattenMeta).foreach(Json.prettyPrint _ andThen println)
{
  "metadata.id" : "1234",
  "metadata.type" : "file",
  "metadata.length" : 395
}

Just change the path if you want to handle metadata fields somewhere else in the tree.

---

Note that using a transformer may be overkill here—see e.g. Pascal Voitot's input in this thread, where he proposes the following:

(json \ "metadata").as[JsObject].fields.foldLeft(Json.obj()) {
  case (acc, (k, v)) => acc + (s"metadata.$k" -> v)
}

It's not as composable, and you'd probably not want to use as in real code, but it may be all you need.

Answer 5

Based on previous solutions, have tried to simplify the code a bit

  def getNewKey(oldKey: String, newKey: String): String = {
    if (oldKey.nonEmpty) oldKey + "." + newKey else newKey
  }

  def flatten(js: JsValue, prefix: String = ""): JsObject = {
    if (!js.isInstanceOf[JsObject]) return Json.obj(prefix -> js)
    js.as[JsObject].fields.foldLeft(Json.obj()) {
      case (o, (k, value)) => {
        o.deepMerge(value match {
          case x: JsArray => x.as[Seq[JsValue]].zipWithIndex.foldLeft(o) {
            case (o, (n, i: Int)) => o.deepMerge(
              flatten(n.as[JsValue], getNewKey(prefix, k) + s"[$i]")
            )
          }
          case x: JsObject => flatten(x, getNewKey(prefix, k))
          case x => Json.obj(getNewKey(prefix, k) -> x.as[JsValue])
        })
      }
    }
  }

Answer 6

Thanks m-z, it is very helpful. (I'm not so familiar with Scala.)

I'd like to add a line for "flatten" working with primitive JSON array like "{metadata: ["aaa", "bob"]}".

  def flatten(js: JsValue, prefix: String = ""): Seq[JsValue] = {

    // JSON primitive array can't convert to JsObject
    if(!js.isInstanceOf[JsObject]) return Seq(Json.obj(prefix -> js))

    js.as[JsObject].fieldSet.toSeq.flatMap{ case (key, values) =>
      values match {
        case JsBoolean(x) => Seq(Json.obj(concat(prefix, key) -> x))
        case JsNumber(x) => Seq(Json.obj(concat(prefix, key) -> x))
        case JsString(x) => Seq(Json.obj(concat(prefix, key) -> x))
        case JsArray(seq) => seq.zipWithIndex.flatMap{ case (x, i) => flatten(x, concat(prefix, key + s"[$i]")) }
        case x: JsObject => flatten(x, concat(prefix, key))
        case _ => Seq(Json.obj(concat(prefix, key) -> JsNull))
      }
    }
  }