How do I set an environment or substitution variab

Question

Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?

Specifically, I'd like to make a custom image tag that's based on a combination of the timestamp and $SHORT_SHA. Something like the below. Though, it doesn't work, as docker complains about "export", and, even if that worked, it likely will be a different env:

  # Setting tag in a variable:
  - name: 'ubuntu'
    args: ['export', '_BUILD_TAG=`date', '-u', '+%Y%m%dT%H%M%S_$SHORT_SHA`']

Then, in a later step:

  # Using tag from the variable:
  - name: gcr.io/cloud-builders/docker
    args: ['build', '-t', 'gcr.io/$PROJECT_ID/$_BUILD_TAG', '.']

So, how do I use the output of one step in another? I could write the contents of date to a file, and then read it, but I'm back at not knowing how to set the variable from the file I read (or otherwise interpolate its results to form the argument to docker build).

Answer 1

I never found a way to set an environment variable in one build step that can be read in other steps, but I ended up accomplishing the same effect by building on Konstantin's answer in the following way:

In an early step, I generate and write my date-based tag to a file. The filesystem (/workspace) is retained between steps, and serves as store of my environment variable. Then, in each step that I need to reference that value, I cat that file in place. The trick is to use sh or bash as the entrypoint in each container so that the sub-shell that reads from the file can execute.

Here's an example:

## Set build tag and write to file _TAG
- name: 'ubuntu'
  args: ['bash', '-c', 'date -u +%Y%m%dT%H%M_$SHORT_SHA > _TAG']

...

# Using the _TAG during Docker build:
- name: gcr.io/cloud-builders/docker
entrypoint: sh
args: ['-c', 'docker build -t gcr.io/$PROJECT_ID/image_name:$(cat _TAG) .']

A caveat to note is that if you are doing the bash interpolation in this way within, say, a JSON object or something that requires double quotes, you need the subshell call to never be surrounded by single quotes when executed in the container, only double, which may require escaping the internal double quotes to build the JSON object. Here's an example where I patch the kubernetes config using the _TAG file value to deploy the newly-build image:

- name: gcr.io/cloud-builders/kubectl
entrypoint: bash
args: ['-c', 'gcloud container clusters get-credentials --zone $$CLOUDSDK_COMPUTE_ZONE $$CLOUDSDK_CONTAINER_CLUSTER ; kubectl patch deployment deployment_name -n mynamespace -p "{\"spec\":{\"template\":{\"spec\":{\"containers\":[{\"name\":\"image_name\",\"image\":\"gcr.io/$PROJECT_ID/image_name:$(cat _TAG)\"}]}}}}}"']
env:
- 'CLOUDSDK_COMPUTE_ZONE=us-central1-b'
- 'CLOUDSDK_CONTAINER_CLUSTER=my-google-proj-cluster-name'

Answer 2

- name: gcr.io/cloud-builders/docker
  entrypoint: sh
  args
    - '-c'
    - 'docker build -t gcr.io/$PROJECT_ID/$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA) .'

Answer 3

Here's an example of what I've just done myself to reuse output from GitVersion in another. It builds on the answer @chetabahana posted.

steps:
- id: 'Gitversion: Unshallow repo'
  name: gcr.io/cloud-builders/git
  args: [fetch, --unshallow]

- id: 'Gitversion: Parse'
  name: gittools/gitversion:latest-linux
  entrypoint: /bin/bash
  args:
    - -c 
    - |
      dotnet /app/GitVersion.dll > /workspace/gitversion.json

- id: 'Gitversion: Env file'
  name: stedolan/jq
  entrypoint: /bin/bash
  args:
    - -c
    - |
      for s in $(cat /workspace/gitversion.json | jq -r "to_entries|map(\"\(.key)=\(.value|tostring)\")|.[]" ); do
        echo "export $s" >> /workspace/gitversion.env
      done

- id: 'Build and push API image'
  name: gcr.io/cloud-builders/docker
  entrypoint: /bin/bash
  args:
    - -c
    - |
      source /workspace/gitversion.env
      docker build -t gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData} example-app
      docker push gcr.io/xxxx/example:$${SemVer}-$${BuildMetaData}

The magic ingredient was the $$ to escape substitution variable so that the build job would not attempt to substitute, leaving it for bash to substitute instead.

Answer 4

Although this doesn't solve your problem, I did want to post this answer since the first sentence of your question is, "Basically, when using Google Cloud Build, how do I read a value that was written in an earlier build step in subsequent steps?". This is how you'd do that.

From the official documentation:

A Volume is a Docker container that is mounted into build steps to persist files across build steps. When Cloud Build runs a build step, it automatically mounts a workspace volume into /workspace. You can specify additional volumes to be mounted into your build steps' containers using the volumes field for your steps.

Here's an example of it implemented from someone who asked this question in a github issue, but to put the date in the volume for later reading by another step:

steps:
- name: 'ubuntu'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        date -u +%Y%m%dT%H%M_$SHORT_SHA > /persistent_volume/file
- name: 'gcr.io/cloud-builders/docker'
  volumes:
  - name: 'vol1'
    path: '/persistent_volume'
  args: ['run', '-v', 'vol1:/data', 'alpine', 'cat', 'data/file']

However, for your particular case, I would just tag it with a subshell command like it's done in this answer here:

$(date -u +%Y%m%dT%H%M%S_$SHORT_SHA)

Answer 5

You don't need to export nor mount a volume in your case.

steps:
- name: 'ubuntu'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        printenv

- name: gcr.io/cloud-builders/docker
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        printenv

It will output

BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Step #0: HOSTNAME=XXXXXXXXXXX
Step #0: BUILDER_OUTPUT=/builder/outputs
Step #0: PWD=/workspace
Step #0: HOME=/builder/home
Step #0: SHLVL=1
Step #0: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #0: _=/usr/bin/printenv
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: HOSTNAME=XXXXXXXXXXX
Step #1: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Step #1: PWD=/workspace
Step #1: SHLVL=1
Step #1: HOME=/builder/home
Step #1: DEBIAN_FRONTEND=noninteractive
Step #1: BUILDER_OUTPUT=/builder/outputs
Step #1: _=/usr/bin/printenv
Finished Step #1

So you can use /workspace or /builder/home, but since we cannot use a variable other than the defined substitution on yaml file then put them as script in the repo like this:

steps:
- name: 'ubuntu'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        bash test.bash

- name: gcr.io/cloud-builders/docker
  entrypoint: 'bash'
  args:
  - '-c'
  - |
        bash result.bash

test.bash

#!/bin/bash
SHORT_SHA=myvar
date -u +%Y%m%dT%H%M_$SHORT_SHA > /workspace/myfile.txt

result.bash

#!/bin/bash
_BUILD_TAG=`cat /workspace/myfile.txt`
echo "the transferred value is: $_BUILD_TAG"

Output:

BUILD
Starting Step #0
Step #0: Pulling image: ubuntu
Step #0: Using default tag: latest
Step #0: latest: Pulling from library/ubuntu
Step #0: Digest: sha256:eb70667a801686f914408558660da753cde27192cd036148e58258819b927395
Step #0: Status: Downloaded newer image for ubuntu:latest
Finished Step #0
Starting Step #1
Step #1: Already have image (with digest): gcr.io/cloud-builders/docker
Step #1: the transferred value is: 20190708T1706_myvar
Finished Step #1
PUSH
DONE