Is boto3 client thread-safe

Question

Is boto3 low level client for S3 thread-safe? Documentation is not explicit about it.

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#client

A similar issue is discussed in Github

https://github.com/boto/botocore/issues/1246

But still there is no answer from maintainers.

Answer 1

From documentation:

Low-level clients are thread safe. When using a low-level client, it is recommended to instantiate your client then pass that client object to each of your threads.

Instantiation of the client is not thread safe while an instance is. To make things work in a multi-threaded environment, put instantiation in a global Lock like this:

boto3_client_lock = threading.Lock()

def create_client():
    with boto3_client_lock:
        return boto3.client('s3', aws_access_key_id='your key id', aws_secret_access_key='your access key')

Answer 2

If you take a look at the Multithreading/Processing documentation for boto3 you can see that they recommend one client per session as there is shared data between instance that can be mutated by individual threads.

It also looks like there's an open github issue for this exact question. https://github.com/boto/botocore/issues/1246

Answer 3

I recently tried using the single boto client instance using concurrent.futures.ThreadPoolExecutor. I run into exceptions coming from boto. I assume the boto client is not thread safe in this case.

The exception I got

  File "xxx/python3.7/site-packages/boto3/session.py", line 263, in client
    aws_session_token=aws_session_token, config=config)
  File "xxx/python3.7/site-packages/botocore/session.py", line 827, in create_client
    endpoint_resolver = self._get_internal_component('endpoint_resolver')
  File "xxx/python3.7/site-packages/botocore/session.py", line 694, in _get_internal_component
    return self._internal_components.get_component(name)
  File "xxx/python3.7/site-packages/botocore/session.py", line 906, in get_component
    del self._deferred[name]