Boto3: A Future or coroutine is required

问题:

I am trying to download 2 files from S3, using boto3, and wait until 2 files are downloaded, will continue to process 2 files

What I have done

async def download_files():
    await client.download_file(const.bucket_name, 'file/name.txt', '/tmp/name.txt')
    print('file 1 downloaded')
    await client.download_file(const.bucket_name, 'file/class.txt', '/tmp/class.txt')
    print('file 2 downloaded')
    return True

def main():
    ...
    loop = asyncio.get_event_loop()
    loop.run_until_complete(download_files())
    loop.close()
    ...
main()

I received an error

A Future or coroutine is required

It's first time that I am using asyncio, please advise me.

回答1:

boto3 isn't asyncio-aware... its functions are blocking, rather than being awaitable. So, the absolute minimum to do here would be to just remove await from the calls to download_file, and it should work.

client.download_file(const.bucket_name, 'file/name.txt', '/tmp/name.txt')
print('file 1 downloaded')
client.download_file(const.bucket_name, 'file/class.txt', '/tmp/class.txt')
print('file 2 downloaded')
return True

However, this would have poor concurrency properties if there were to be any other concurrent tasks in the thread's event loop: they would be blocked and wouldn't proceed during the downloads [this would make the usage of asyncio a bit unnecessary... concurrent tasks proceeding is sort of the point of asyncio...]

To have better concurrency properties, you should be able to call the functions via run_in_executor, which by default will run the passed function in another thread.

async def download_files(loop):
  await loop.run_in_executor(None, client.download_file, const.bucket_name, 'file/name.txt', '/tmp/name.txt')
  print('file 1 downloaded')
  await loop.run_in_executor(None, client.download_file, const.bucket_name, 'file/class.txt', '/tmp/class.txt')
  print('file 2 downloaded')
  return True

Alternatively, rather than using boto3 and threads, you could use aiohttp, and roll-you-own AWS authentication (full disclosure: the post on rolling-your-own AWS authentication was written by me)

回答2:

look into aioboto3. Also for more low-level look into aiobotocore. Both available via pip