Stream transformations in AWS Lambda result in wri

Question

I am trying to transform a CSV file using Fastcsv nodejs package. Interestingly, the code snippet works locally on my system well. However, if I try to integrate it with AWS lambda program it gives write after end error.

var stream = fs.createReadStream(s3EventInfo.inputDownloadLoc)
.pipe(csv.parse({headers: true}))
//pipe the parsed input into a csv formatter
.pipe(csv.format({headers: true}))
//Using the transfrom function from the formatting stream
.transform(function(row, next){
    transformLine(row, next);
})
.pipe(fs.createWriteStream(s3EventInfo.outputFileLoc))
.on("end", function(){
    callback();
});

Here is the error in aws logs..

Error: write after end
at writeAfterEnd (_stream_writable.js:133:12)
at PassThrough.Writable.write (_stream_writable.js:181:5)
at write (_stream_readable.js:602:24)
at flow (_stream_readable.js:611:7)
at _stream_readable.js:579:7
at process._tickDomainCallback (node.js:486:13)

Please help in understanding and resolving the issue.

Answer 1

Lambda functions are deployed and run on demand, but additional calls to the same lambda function MAY run on the existing instance if it has not been destroyed. You can't control this but you need to ensure that you code can correctly handle it.

If your stream is being defined in the global scope then your problem is that those subsequent calls are reusing the stream that has already received an 'end' event.

You need to encapsulate the streams so that they are instantiated new for each call.

Answer 2

Your var stream call should be placed inside of your Handler function rather than being declared as a global. The result of this is to scope it to the event such that if the Lambda function is still active and called multiple times, each event will start and end its own stream, rather than sharing a stream that the first event could close before the second event is done using it. I had your exact same errors and was able to resolve it perfectly with this change.