Downloading folders from Google Cloud Storage Buck

Question

I need to download folders with NodeJS from my Bucket from my Google Cloud Storage. I read all the documentation and I only found a way to download files and not folders. I need to get/download the folder to provide user's download files.

Could someone help me?

Answer 1

You will want to use the getFiles method of Bucket to query for the files you want to download, then download each one of them individually. Read more about how to use the underlying list API. There are no folder operations in Cloud Storage (as there are not actually any folders, there are just file paths the look like they're organized as folders).

Answer 2

As Doug said, Google Cloud Storage would show you the structure of different directories, but there are actually no folders within the buckets.

However, you can find perform some workarounds within your code to create that very same folder structure yourself. For the workaround I came up with, you need to use libraries such as shelljs, which will allow you to create folders in your system.

Following this GCP tutorial on Cloud Storage, you will find examples on, for instance, how to list or download files from your bucket.

Now, putting all this together, you can get the full path of the file you are going to download, parse it to separate the folders from the actual file, then create the folder structure using the method mkdir from shelljs.

For me, modifying the method for downloading files in the tutorial, was something like this:

var shell = require('shelljs');
[...]
async function downloadFile(bucketName, srcFilename, destFilename) {
  // [START storage_download_file]
  // Imports the Google Cloud client library
  const {Storage} = require('@google-cloud/storage');

  // Creates a client
  const storage = new Storage();

  //Find last separator index
  var index = srcFilename.lastIndexOf('/');
  //Get the folder route as string using previous separator
  var str = srcFilename.slice(0, index);
  //Create recursively the folder structure in the current directory
  shell.mkdir('-p', './'+str);
  //Path of the downloaded file
  var destPath = str+'/'+destFilename;

  const options = {
    destination: destPath,
  };

  // Downloads the file
  await storage
    .bucket(bucketName)
    .file(srcFilename)
    .download(options);

  console.log(
    `gs://${bucketName}/${srcFilename} downloaded to ${destPath}.`
  );
  // [END storage_download_file]
}