Call Solr asynchronous from Play Framework

Question

I have created a Play 2.1 Scala application. I am uncertain what's the best way to call Solr from a Play application:

• There is no Solr module for Play 2.
• AFAIK all Solr-APIs like SolrJ are blocking.
• I could wrap a SolrJ call into a Future, but this will also block a thread, correct?

• Should I use the play.api.libs.ws.WS library to call Solr and use Plays JSON support to extract the result (like in the example below) or is there any easier/faster way?

  val solrQuery: Future[play.api.libs.ws.Response] = WS.url("http://localhost:8983/solr/collection1/select?q=id%3A123&wt=json").get()
  

Answer 1

Came across this need recently and didn't find anything useful googling about. Below is only for querying but could be expanded. I'm assuming you want to stay with SolrJ classes. The SolrQuery and QueryResponse are pretty easy to work with.

So to query. You'll want to build up your SolrQuery as normal. For "wt" supply "javabin". This will give you a response in the compressed binary format that SolrJ uses internally.

val sq = new SolrQuery()
sq.set("wt", "javabin")
...

You'll want to turn your SolrQuery into something that WS understands. (I haven't added all the imports since most are straightforward to figure out [e.g., by your IDE]. Those I have included might not be as obvious.)

import scala.collection.JavaConverters._

def solrQueryToForm(sq: SolrQuery): Map[String, Seq[String]] = {
  sq.getParameterNames.asScala.foldLeft(Map.empty[String, Seq[String]]) {
    case (m, n) =>
      m + (n -> sq.getParams(n))
  }
}

In my shop we use a default collection and handler (i.e., "/select") but you'll want those to be overridden by the SolrQuery

def solrEndpoint(sq: SolrQuery): String = {
  val coll = sq.get("collection", defaultCollection)
  val hand = Option(sq.getRequestHandler).getOrElse(defaultHandler)
  formSolrEndpoint(solrUrl, coll, hand)
}

def formSolrEndpoint(base: String, collection: String, handler: String): String = {
  val sb = new StringBuilder(base)
  if (sb.last != '/') sb.append('/')
  sb.append(collection)
  if (!handler.startsWith("/")) sb.append('/')
  sb.append(handler)
  sb.result()
}

You'll need some code to map the WSResponse to a QueryResponse

import com.ning.http.client.{Response => ACHResponse}

def wsResponseToQueryResponse(wsResponse: WSResponse)(implicit ctx: ExecutionContext): QueryResponse = {
  val jbcUnmarshal = {
    val rbis = wsResponse.underlying[ACHResponse].getResponseBodyAsStream

    try {
      new JavaBinCodec().unmarshal(rbis)
    }
    finally {
      if (rbis != null)
        rbis.close()
    }
  }

  // p1: SolrJ pulls the same cast
  // p2: We didn't use a SolrServer to chat with Solr so cannot provide it to QueryResponse
  new QueryResponse(jbcUnmarshal.asInstanceOf[NamedList[Object]], null)
}

And that gives you all the pieces to call Solr using Play's async WS service.

def query(sq: SolrQuery)(implicit ctx: ExecutionContext): Future[QueryResponse] = {
  val sqstar = sq.getCopy
  sqstar.set("wt", "javabin")

  WS.url(solrEndpoint(sqstar))
    .post(solrQueryToForm(sqstar))
    .map(wsResponseToQueryResponse)
}

Since Play now publishes the webservice code as a standalone jar this means pretty much any project should be able to query Solr asynchronously. Hope that's useful.

Answer 2

Here's how I use WS in my side project:

val itselfNodeFuture = Statix.doParams( Statix.SolrSelectWSReq, 
    List(
    "wt"     -> "json", 
    "q"      -> "*:*",
    "fq"     -> "node_type:collection",
    "fq"     -> "id:%d".format( nodeId),
    "indent" -> "true",
    "rows"   -> "1",
    "fl"     -> "id,parent_id,title",
    "fl"     -> "date_created,date_about,date_modified")
).get()

//Use the first Await after the last future
val itselfJson = Await.result(
    itselfNodeFuture, Duration("2 sec")).json

val mainRow = (itselfJson \ "response" \ "docs").as[ Seq[JsValue]]
val mainNodeParent = (mainRow(0) \ "parent_id").as[Long]
val mainNodeTitle = (mainRow(0) \ "title").as[String]

And here's the utility class I use, the doParams is especially useful.

object Statix { //Noder must extend this
    def SolrSelectWSReq = WS.url("http://127.0.0.1:8080/solr-store/collection1/select/")
    def SolrUpdateWSReq = WS.url("http://127.0.0.1:8080/solr-store/collection1/update/json/")

    def doParams(request: WS.WSRequestHolder, params: List[(String, String)]) = {
        params.foldLeft( request){
            (wsReq, tuple) => wsReq.withQueryString( tuple)}}
}

Answer 3

You want to wrap the call in a Future with its own Execution context. This way the call may be blocking, but it will use a different thread pool, not blocking the main application.

In fact, this is standard behaviour when facing blocking or slow tasks, like sending queries to a database or doing some heavy-lifting task.