Streaming huge json with Akka Stream

2019-06-01 08:33发布

问题:

I have a problem of huge http response with a json slab, where only portion is point of interest. I cannot change the response structure. here is an example

{
  "searchString": "search",
  "redirectUrl": "",
  "0": {
    "numRecords": 123,
    "refinementViewModelCollector": {},
//    Lots of data here
    "results": [
      {
        "productCode": "123",
        "productShortDescription": "Desc",
        "brand": "Brand",
        "productReview": {
          "reviewScore": 0
        },
        "priceView": {
          "salePriceDisplayable": false,
        },
        "productImageUrl": "url",
        "alternateImageUrls": [
          "url1"
        ],
        "largeProductImageUrl": "url4",
        "videoUrl": ""
      },
      {
        "productCode": "124",
        "productShortDescription": "Desc",
        "brand": "Brand",
        "productReview": {
          "reviewScore": 0
         },
        "priceView": {
          "salePriceDisplayable": false,
        },
        "preOrder": false,
        "productImageUrl": "url",
        "alternateImageUrls": [
          "url1"
        ],
        "largeProductImageUrl": "url4",
        "videoUrl": ""
      }
    ]
    //lots of data here
  }
}

My point of interest is entries in results Jason Array, but the are sitting in the middle of json

I created a small Play WS Client like this:

val wsClient: WSClient = ???
val ret = wsClient.url("url").stream()
ret.flatMap { response =>
  response.body.via(JsonFraming.objectScanner(1024))
    .map(_.utf8String)
    .runWith(Sink.foreach(println))
}

this will not work because it will take whole json slab as Json object. I need to skip some data until "results": entry appear in the stream, then start parsing entries and skip all the rest. Any ideas how to do this?

回答1:

There are parsers that support parsing as a stream. For a good example check out this Circe example https://github.com/circe/circe/tree/master/examples/sf-city-lots