Dynamically change branches on AWS CodePipeline

Question

I am looking for a good solution on how to run parametrized (customized) builds in CodePipeline where branch can be changed dynamically?

A little background on the problem: I need an on-demand environment that will be started on certain branch. We already use Bamboo CI server for part of the infrastructure and this is easily achievable with customized build also in Jenkins.

So basically I need a way to trigger a build with branch as a variable on CodePipeline in AWS.

Answer 1

We can very well have dynamic branching support with the following approach.

We follow this approach in our organisation and it works very well.

One of the limitations in AWS code-pipeline is that we have to specify branch names while creating the pipeline. We can however overcome this issue using the architecture shown below.

flow diagram

Create a Lambda function which takes the GitHub web-hook data as input, using boto3 integrate it with AWS pipeline(pull the pipeline and update), have an API gateway to make the call to the Lambda function as a rest call and at last create a web-hook to the GitHub repository.

External links:

• https://aws.amazon.com/quickstart/architecture/git-to-s3-using-webhooks/

• https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/codepipeline.html

Answer 2

Currently CodePipeline does not support branch based builds. Typically CodePipeline works best for running validations and automating the release of your "release" branch.

One option for pre merge validation is to use CodeBuild pull request support to validate pull requests then use CodePipeline to validate the merged code: https://aws.amazon.com/about-aws/whats-new/2017/09/aws-codebuild-now-supports-building-github-pull-requests/

Answer 3

For your use case it is best to create a pipeline specifically for each branch, as it sounds like your branch will have a fixed name for a given environment.

This works well where branches represent environments, where CodePipeline struggles is performing Continuous Delivery for more dynamic branches, such as feature branches/pull requests.

For the latter scenario I use CodeBuild to process pull requests, and then publish build artefacts in an S3 archive that I then use to trigger CodePipeline to run integration tests and staging deployments. There are a few traps along the way, but it allows you to leverage some of the more powerful features of CodePipeline (e.g. ability to only have a single stage execution running at a time, which is important for environments with shared resources).