Is boto3 low level client for S3 thread-safe? Documentation is not explicit about it.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#client
A similar issue is discussed in Github
https://github.com/boto/botocore/issues/1246
But still there is no answer from maintainers.
If you take a look at the Multithreading/Processing documentation for boto3 you can see that they recommend one client per session as there is shared data between instance that can be mutated by individual threads.
It also looks like there's an open github issue for this exact question. https://github.com/boto/botocore/issues/1246
I recently tried using the single boto client instance using concurrent.futures.ThreadPoolExecutor. I run into exceptions coming from boto. I assume the boto client is not thread safe in this case.
The exception I got
File "xxx/python3.7/site-packages/boto3/session.py", line 263, in client
aws_session_token=aws_session_token, config=config)
File "xxx/python3.7/site-packages/botocore/session.py", line 827, in create_client
endpoint_resolver = self._get_internal_component('endpoint_resolver')
File "xxx/python3.7/site-packages/botocore/session.py", line 694, in _get_internal_component
return self._internal_components.get_component(name)
File "xxx/python3.7/site-packages/botocore/session.py", line 906, in get_component
del self._deferred[name]
From documentation:
Low-level clients are thread safe. When using a low-level client, it is recommended to instantiate your client then pass that client object to each of your threads.
Instantiation of the client is not thread safe while an instance is. To make things work in a multi-threaded environment, put instantiation in a global Lock like this:
boto3_client_lock = threading.Lock()
def create_client():
with boto3_client_lock:
return boto3.client('s3', aws_access_key_id='your key id', aws_secret_access_key='your access key')
