Asynchronous POST Requests - R, using RCurl?

Question

I am trying to make async requests to a REST API from R. The below curl command illustrates the parameters that I need to the pass to the api. I'm giving you guys the linux curl command as I'm hoping that will make it clear:

curl -v -X POST https://app.example.com/api/ \
-H 'Authorization: somepwd' \
-H "Content-Type: application/json" \
-d {key1: value1, key2: value2}

Right now, I'm accomplishing the same thing in R by executing the following:

library(httr)
library(jsonlite)
content(POST('https://app.example.com/api/'
                    ,add_headers(Authorization = 'somepwd') 
                    ,body = toJSON(rDataFrame)
                    ,content_type_json()
             )
        )

The goal is to submit the above POST request from R but to vary the json string that is sent in the body, and do that asynchronously.

I have been searching for packages that will help me make asynchronous requests rather than making requests serially. The closest thing I could find is the getURIAsynchronous() function from the RCurl package (https://cran.r-project.org/web/packages/RCurl/RCurl.pdf) but do not understand how to submit a PUT request with headers and a body using their function. I would really like to make the above POST request from R but asynchronously where the URI is the same, but the data sent is different for each request.

I found this http://www.omegahat.org/RCurl/concurrent.html

getURIs =
function(uris, ..., multiHandle = getCurlMultiHandle(), .perform = TRUE)
{
  content = list()
  curls = list()

  for(i in uris) {
    curl = getCurlHandle()
    content[[i]] = basicTextGatherer()
    opts = curlOptions(URL = i, writefunction = content[[i]]$update, ...)    
    curlSetOpt(.opts = opts, curl = curl)
    multiHandle = push(multiHandle, curl)
  }

  if(.perform) {
     complete(multiHandle)
     lapply(content, function(x) x$value())
   } else {
     return(list(multiHandle = multiHandle, content = content))
   }
} 

My idea is that I could replace for (i in uris) with for(i in jsons) where I am looping over the different data that I want to send to the same URL, however I am having trouble understanding the following concepts from the RCurl Package:

1. How do I pass a header as part of a PUT request. How do I pass data in the body of the request? This was pretty straight forward using the httr package as I have illustrated above.

2. I tried passing in the header in the curl options and alternatively the header. The thing is I don't understand where to pass the the component parts of the post request: authentication, header, and body within the getURIAsynchronous() function, or any of the resources I have described above.

Does anyone know how to accomplish this? An example would be incredibly helpful.

Answer 1

The curl package has been recently updated to handle async requests (see here)

Using the curl, magrittr and jsonlite packages you can create asynchronous post requests by:

• Creating a generic handle with your header and body content using the handle_setform function
• Writing a call back function to retrieve your results
• Initializing a pool and adding your concurrent requests to it
• Running your pool via multi_run

Example code is below:

library(curl)
library(jsonlite)
library(magrittr)

#create a handle object
h <- new_handle() %>%
handle_setheaders(Authorization = "somepwd",
                  "Content-Type" = "application/json") %>%
  handle_setform(body = toJSON(iris))

pool <- new_pool()
# results only available through call back function
cb <- function(req){cat("done:", req$url, ": HTTP:", req$status, "\n", "content:", rawToChar(req$content), "\n")}

# example vector of uris to loop through
uris <- c("https://app.example.com/api/endpoint1"
          ,"https://app.example.com/api/endpoint2"
          ,"https://app.example.com/api/endpoint3")

# all scheduled requests are performed concurrently
sapply(uris, curl_fetch_multi, done=cb, pool=pool)

# This actually performs requests
out <- multi_run(pool = pool)