How can I bypass the 10MB limit of AWS API gateway

Question

what I want

An API which takes file and some parameters using POST and gives back a JSON response.

curl -X POST www.endpoint.com \
   -F file=@/myfile.txt \
   -F foo=bar # other params

I have this working with Lambda + API gateway using binary data but 10MB limit is the issue.

I have considered a POST API which uploads file to S3. The event generated is then read by Lambda. But for this I have few questions-

1. Where will my other parameters go?
2. How will Lambda return back the response?

Answer 1

Have an endpoint on your API gateway that generates pre-signed URLs for uploading files to S3. Then your client application can invoke that endpoint to get the pre-signed URL, after which it can upload the file to S3. Then you can have a Lambda function that is triggered by new objects in your S3 bucket that will read the file from S3 and process it.

To post extra parameters along with the file you have a few options:

• Option 1: Post the extra parameters along with the initial request for the pre-signed URL. Have the Lambda function that generates the pre-signed URL store those parameters somewhere like DynamoDB, along with the S3 object key the pre-signed URL is being generated for. Then when the other Lambda function is triggered by the new object appearing in S3 it can look up those extra parameters from DynamoDB.

• Option 2: When uploading the file to S3 via the pre-signed URL, your application can add extra header fields to the upload that will be stored on the S3 object as metadata.

Answer 2

Your use case simply isn't possible with one API call if you want to stick with a serverless solution.

A possible serverless solution would be a 3 step process for the client.

Step 1

Call api1 to get a signed url for S3. This would point to a Lambda that creates a UUID and uses that UUID to construct a signed URL for S3 (i.e. uses the UUID as the filename of the file being received). The response would be the URL and the UUID.

Step 2

PUT file to s3 using the signed URL.

Step 3

Call api2 and pass the UUID and what ever other parameters are required. This api also points to a Lambda which now knows where the file is (thanks to the UUID) and has whatever other parameters are required to process the file and give a response.