I wrote the following YAML file , to use the gitlab CI runner:
variables:
GCP_PROJECT_ID: project
CLUSTER_NAME: cluster
DAGS_BUCKET : gs://bucket/dags
PLUGINS_BUCKET : gs://bucket/plugins
JARS_BUCKET: gs://bucket/jars
COMPOSER_ENVIRONMENT: test
ENVIRONMENT_LOCATION: us
FIRST_DAG: first-dag
SECOND_DAG: second-dag
CHECKPOINT_FOLDER: gs://bucket/checkpoints
BIGQUERY_DATASET: test_dataset
.tags_fdedtt:
tags:
- gcpd
- ded
stages:
- deploy
deploy:
stage: deploy
image: google/cloud-sdk
script:
- echo $KEY_FILE_CONTENT > key.json
- gcloud auth activate-service-account --key-file=key.json
- gcloud beta composer environments storage dags delete --environment ${COMPOSER_ENVIRONMENT} --project ${GCP_PROJECT_ID} --location ${ENVIRONMENT_LOCATION}
- chmod +x variables.sh
- ./variables.sh ${CLUSTER_NAME} ${GCP_PROJECT_ID} ${JARS_BUCKET} ${FIRST_DAG} ${COMPOSER_ENVIRONMENT} ${ENVIRONMENT_LOCATION} ${CHECKPOINT_FOLDER} ${BIGQUERY_DATASET}
- gsutil cp plugins/*.py ${PLUGINS_BUCKET}
- gsutil cp dags/*.py ${DAGS_BUCKET}
- sleep 2m
- gcloud beta composer environments run ${COMPOSER_ENVIRONMENT} --project ${GCP_PROJECT_ID} --location ${ENVIRONMENT_LOCATION} trigger_dag -- ${FIRST_DAG}
As you may notice the variables.sh file takes a lot of arguments in input , I want to find a more sophisticated way to pass variables to the variables.sh script
otherwise the line ./variables.sh will be really huge. Do you have any idea how to solve this in a stylish way ?
