How to increase Java Heap size in aws Lambda?

2019-09-10 21:37发布

问题:

I am using AWS SDK http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/s3/package-summary.html

to program a data processing in AWS Lambda

encountered the error

    {
  "errorMessage": "Java heap space",
  "errorType": "java.lang.OutOfMemoryError",
  "stackTrace": [
    "java.util.Arrays.copyOf(Arrays.java:3332)",
    "java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)",
    "java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)",
    "java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:569)",
    "java.lang.StringBuffer.append(StringBuffer.java:369)",
    "java.io.StringWriter.write(StringWriter.java:94)",
    "org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1129)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1104)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1050)",
    "org.apache.commons.io.IOUtils.copy(IOUtils.java:1075)",
    "org.apache.commons.io.IOUtils.toString(IOUtils.java:382)",
    "example.Hello.myHandler(Hello.java:47)",
    "sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)",
    "sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)",
    "sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)",
    "java.lang.reflect.Method.invoke(Method.java:497)"
  ]
}

I want to solve this by increase heap size programmatically

The code caused this problem is

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
String theString = IOUtils.toString(objectData, null); 

The object I am extracting is about 300MB. and Configuration of Lambda

Memory: 1024 MB

回答1:

You will be better off using Gson or Jackson APIs to parse your Json object rather than working with Lambda limitation.

something like that can help you started

AmazonS3 s3Client = new AmazonS3Client();
S3Object s3Object = s3Client.getObject(new GetObjectRequest(srcBucket, srcKey));
InputStream objectData = s3Object.getObjectContent();

// transfer data into JSON
JsonFactory jsonfactory = new JsonFactory(); 
JsonParser parser = jsonfactory.createJsonParser(objectData); 

// starting parsing of JSON String 
while (parser.nextToken() != JsonToken.END_OBJECT) { 
    // do what you need to do
}