Unable to create new native thread error when invo

Question

I have a lambda function that queries some DynamoDB tables and returns some entries in them. Invoking it a few times and all is fine, but I am now stress testing my back end and when I invoke this function over and over in a short period of time, I get the following error:

"errorMessage":"unable to create new native thread","errorType":"java.lang.OutOfMemoryError"

I am using DynamoDB's spatial indexing capabilities, and the error message seems to be suggesting that a line to do with the spatial index query is causing the problems. These are the lines to do with the geo querying:

GeoDataManagerConfiguration config = new GeoDataManagerConfiguration(ddb, "geo");

        GeoDataManager geoIndexManager = new GeoDataManager(config);

        GeoPoint centerPoint = new GeoPoint(request.latitude, request.longitude);

        QueryRadiusRequest queryRadiusRequest = new QueryRadiusRequest(centerPoint, request.radius);

        QueryRadiusResult queryRadiusResult = geoIndexManager.queryRadius(queryRadiusRequest);

Specifically though the following line seems to be the root of the problems:

QueryRadiusResult queryRadiusResult = geoIndexManager.queryRadius(queryRadiusRequest); 

What is going on? Why does this line throw this error when the function is being invoked repeatedly? At face value it seems like spatial indexing/look ups on dynamoDB can't handle being invoked in high numbers, which really doesn't make sense, surely when it was built it was made to handle scale. Is there some setting where i can allocate more memory in aws to avoid this problem?

Answer 1

make your GeoDataManagerConfiguration static and don't create it for every request (in your lambda handler). You will create an execute service for every request otherwise: https://github.com/awslabs/dynamodb-geo/blob/8fd3e7b408650a937735ffb6527516dba239a1db/src/com/amazonaws/geo/GeoDataManagerConfiguration.java#L164

Answer 2

See AWS Lambda Limits.

You have a limit of 1024 threads per invocation, and a maximum of 100 invocations per region.

If you're hitting that limit in your single function, you could set a hard limit below that in your code, or add a thread pool and a queue.

I've previously written a Lambda function to read in a config file and parcel test data out to other copies of itself, to get around such limits.