How can i download the files inside a folder on go

Question

from google.cloud import storage
client = storage.Client()
bucket = client.get_bucket([bucket_name])
blob = bucket.get_blob([path to the .txt file])
blob.download_to_filename([local path to the downloaded .txt file])

How can i adjust my python code to add something like for filename in os.listdir(path): to just copy all the files in a certain folder on there locally

Answer 1

First of all, I think it is interesting to highlight that Google Cloud Storage uses a flat name space, and in fact the concept of "directories" does not exist, as there is no hierarchical file architecture being stored in GCS. More information about how directories work can be found in the documentation, so it is a good read if you are interested in this topic.

That being said, you can use a script such as the one I share below, in order to download all files in a "folder" in GCS to the same folder in your local environment. Basically, the only important addition a part from your own code is that the bucket.list_blobs() method is being called, with the prefix field pointing to the folder name, in order to look for blobs which only match the folder-pattern in their name. Then, you iterate over them, discard the directory blob itself (which in GCS is just a blob with a name ending in "/"), and download the files.

from google.cloud import storage
import os

# Instantiate a CGS client
client=storage.Client()
bucket_name= "<YOUR_BUCKET_NAME>"

# The "folder" where the files you want to download are
folder="<YOUR_FOLDER_NAME>/"

# Create this folder locally
if not os.path.exists(folder):
    os.makedirs(folder)

# Retrieve all blobs with a prefix matching the folder
bucket=client.get_bucket(bucket_name)
blobs=list(bucket.list_blobs(prefix=folder))
for blob in blobs:
    if(not blob.name.endswith("/")):
        blob.download_to_filename(blob.name)