How do you structure sequential AWS service calls

Question

I'm coming from a java background so a bit of a newbie on Javascript conventions needed for Lambda.

I've got a lambda function which is meant to do several AWS tasks in a particular order, depending on the result of the previous task.

Given that each task reports its results asynchronously, I'm wondering if the right way make sure they all happen in the right sequence, and the results of one operation are available to the invocation of the next function.

It seems like I have to invoike each function in the callback of the prior function, but seems like that will some kind of deep nesting and wondering if that is the proper way to do this.

For example on of these functions requires a DynamoDB getItem, following by a call to SNS to get an endpoint, followed by a SNS call to send a message, followed by a DynamoDB write.

What's the right way to do that in lambda javascript, accounting for all that asynchronicity?

Answer 1

Just saw this old thread. Note that future versions of JS will improve that. Take a look at the ES2017 async/await syntax that streamlines an async nested callback mess into a clean sync like code. Now there are some polyfills that can provide you this functionality based on ES2016 syntax.

As a last FYI - AWS Lambda now supports .Net Core which provides this clean async syntax out of the box.

Answer 2

By default Javascript is asynchronous.

So, everything that you have to do, it's not to use those libraries, you can, but there's simple ways to solve this. In this code, I sent the email, with the data that comes from the event, but if you want, you just need to add more functions inside functions.

What is important is the place where your context.done(); is going to be, he is going to end your Lambda function. You need to put him in the end of the last function.

var AWS = require('aws-sdk');    
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';
var ses = new AWS.SES({apiVersion: '2010-12-01'});

exports.handler = function(event, context) {

    console.log(event.nome);
    console.log(event.email);
    console.log(event.mensagem);

    nome = event.nome;
    email = event.email;
    mensagem = event.mensagem;

    var to = ['[e-mail]'];
    var from = '[e-mail]';

    // Send email
    mensagem = ""+nome+"||"+email+"||"+mensagem+"";

    console.log(mensagem);
    ses.sendEmail( { 
       Source: from, 
       Destination: { ToAddresses: to },
       Message: {
           Subject: {
              Data: 'Form contact our Site'
           },
           Body: {
               Text: {
                   Data: mensagem,
               }
            }
       }
    },
    function(err, data) {
        if (err) {
            console.log("ERROR="+err, err.stack); 
            context.done();
          } else {
            console.log("EMAIL SENT="+data);
            context.done();
          }
     });
}

Answer 3

I like the answer from @jonathanbaraldi but I think it would be better if you manage control flow with Promises. The Q library has some convenience functions like nbind which help convert node style callback API's like the aws-sdk into promises.

So in this example I'll send an email, and then as soon as the email response comes back I'll send a second email. This is essentially what was asked, calling multiple services in sequence. I'm using the then method of promises to manage that in a vertically readable way. Also using catch to handle errors. I think it reads much better just simply nesting callback functions.

var Q = require('q');
var AWS = require('aws-sdk');    
AWS.config.credentials = { "accessKeyId": "AAAA","secretAccessKey": "BBBB"};
AWS.config.region = 'us-east-1';

// Use a promised version of sendEmail
var ses = new AWS.SES({apiVersion: '2010-12-01'});
var sendEmail = Q.nbind(ses.sendEmail, ses);

exports.handler = function(event, context) {

    console.log(event.nome);
    console.log(event.email);
    console.log(event.mensagem);

    var nome = event.nome;
    var email = event.email;
    var mensagem = event.mensagem;

    var to = ['[e-mail]'];
    var from = '[e-mail]';

    // Send email
    mensagem = ""+nome+"||"+email+"||"+mensagem+"";

    console.log(mensagem);

    var params = { 
        Source: from, 
        Destination: { ToAddresses: to },
        Message: {
        Subject: {
            Data: 'Form contact our Site'
        },
        Body: {
            Text: {
                Data: mensagem,
            }
        }
    };

    // Here is the white-meat of the program right here.
    sendEmail(params)
        .then(sendAnotherEmail)
        .then(success)
        .catch(logErrors);

    function sendAnotherEmail(data) {
        console.log("FIRST EMAIL SENT="+data);

        // send a second one.
        return sendEmail(params);
    }

    function logErrors(err) {
        console.log("ERROR="+err, err.stack);
        context.done();
    }

    function success(data) {
        console.log("SECOND EMAIL SENT="+data);
        context.done();
    }
}

Answer 4

A very specific solution that comes to mind is cascading Lambda calls. For example, you could write:

1. A Lambda function gets something from DynamoDB, then invokes…
2. …a Lambda function that calls SNS to get an endpoint, then invokes…
3. …a Lambda function that sends a message through SNS, then invokes…
4. …a Lambda function that writes to DynamoDB

All of those functions take the output from the previous function as input. This is of course very fine-grained, and you might decide to group certain calls. Doing it this way avoids callback hell in your JS code at least.

(As a side note, I'm not sure how well DynamoDB integrates with Lambda. AWS might emit change events for records that can then be processed through Lambda.)

Answer 5

Short answer:

Use Async / Await — and Call the AWS service (SNS for example) with a .promise() extension to tell aws-sdk to use the promise-ified version of that service function instead of the call back based version.

Since you want to execute them in a specific order you can use Async / Await assuming that the parent function you are calling them from is itself async.

For example:

let snsResult = await sns.publish({
    Message: snsPayload,
    MessageStructure: 'json',
    TargetArn: endPointArn
}, async function (err, data) {
    if (err) {
        console.log("SNS Push Failed:");
        console.log(err.stack);
        return;
    }
    console.log('SNS push suceeded: ' + data);
    return data;
}).promise();

The important part is the .promise() on the end there. Full docs on using aws-sdk in an async / promise based manner can be found here: https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/using-promises.html

In order to run another aws-sdk task you would similarly add await and the .promise() extension to that function (assuming that is available).

For anyone who runs into this thread and is actually looking to simply push promises to an array and wait for that WHOLE array to finish (without regard to which promise executes first) I ended up with something like this:

let snsPromises = [] // declare array to hold promises
let snsResult = await sns.publish({
    Message: snsPayload,
    MessageStructure: 'json',
    TargetArn: endPointArn
}, async function (err, data) {
    if (err) {
        console.log("Search Push Failed:");
        console.log(err.stack);
        return;
    }
    console.log('Search push suceeded: ' + data);
    return data;
}).promise();

snsPromises.push(snsResult)
await Promise.all(snsPromises)

Hope that helps someone that randomly stumbles on this via google like I did!

Answer 6

I found this article which seems to have the answer in native javascript.

Five patterns to help you tame asynchronis javascript.