Azure CLI : bash script to upload files in paralle

Question

Is there a Azure CLI upload option to parallel upload files to blob storage. There is folder with lots of files. Currently the only option I have is do a for loop with below command and upload is sequentially.

az storage blob upload --file $f --container-name $CONTAINERNAME --name $FILEINFO

Answer 1

For now, it is not possible. With the Azure CLI 2.0 there is no option or argument to upload the contents of a specified directory to Blob storage recursively. So, Azure CLi 2.0 does not support upload files in parallel.

If you want to upload multiple files in parallel, you could use Azcopy.

AzCopy /Source:C:\myfolder /Dest:https://myaccount.blob.core.windows.net/mycontainer /DestKey:key /S

Specifying option /S uploads the contents of the specified directory to Blob storage recursively, meaning that all subfolders and their files will be uploaded as well.

As you mentioned, you could use loop to upload files, but it does not support upload files in parallel. Try following script.

export AZURE_STORAGE_ACCOUNT='PUT_YOUR_STORAGE_ACCOUNT_HERE'
export AZURE_STORAGE_ACCESS_KEY='PUT_YOUR_ACCESS_KEY_HERE'

export container_name='nyc-tlc-sf'
export source_folder='/Volumes/MacintoshDisk02/Data/Misc/NYC_TLC/yellow/2012/*'
export destination_folder='yellow/2012/'

#echo "Creating container..."
#azure storage container create $container_name

for f in $source_folder
do
  echo "Uploading $f file..."
  azure storage blob upload $f $container_name $destination_folder$(basename $f)
done

echo "List all blobs in container..."
azure storage blob list $container_name

echo "Completed"