Kubernetes CI/CD pipeline

Question

My company has decided to transition to a micro/service based architecture.

We have been doing a bunch of research for the last couple of months on exactly what the architecture of this thing is going to look like.

So far, we've settled on:

• Dotnet core for service development (although being language agnostic is somewhat of an end goal)

• Kafka for message brokering

• Docker

• Kubernetes

• Ansible

We have a pretty basic proof of concept working, which seems to have ticked all the right boxes with the management team, and is an absolute joy to work with.

My next task is to investigate options for how the development workflow is actually going to work. They are already used to working in a CI/CD manner, with some of their newer products using Jenkins/Octopus Deploy.

My question is: Do any of you have any firm recommendations for setting up a CI/CD pipeline when deploying to a Kubernetes cluster?

A list of must haves is:

• Multiple environments i.e. Integration, Test, UAT, Staging, Production.

• A means through which different business units can uniquely handle deployments to different environments (development can only push to integration, tester into test, etc). This one is probably their biggest ask - they are used to working with Octopus, and they love the way it handles this.

• The ability to roll back / deploy at the click of a button (or with as few steps as possible).

We would be deploying to our own servers initially.

I've spent the past couple of days looking in to options, of which there are many.

So far, Jenkins Pipeline seems like it could be a great start. Spinnakar also seems like a solid choice. I did read a bit into Fabric8, and while it offers much of what I'm asking, it seems a bit like overkill.

Answer 1

If you want to use Jenkins, Pipelines are indeed the way to go. Our setup does pretty much what you want, so let me explain how we set it up.

We use a Jenkins agent that has docker and kubectl installed. This agent first builds the docker container and pushes it to our docker registry. It will then call kubectl in various stages to deploy to our testing, acceptance and production clusters.

Different business units: in a Pipeline you can use an input step to ask whether the Pipeline should proceed or not. You can specify who may press the button, so this is how you could solve the deployment to different clusters. (Ideally, when you get to CD, people will realize that pressing the button several times per day is silly and they'll just automate the entire deployment.)

Rollback: we rely on Kubernetes's rollback system for this.

Credentials: we provision the different Kubernetes credentials using Ansible directly to this Jenkins agent.

To reduce code duplication, we introduced a shared Jenkins Pipeline library, so each (micro)service talks to all Kubernetes clusters in a standardized way.

Note that we use plain Jenkins, Docker and Kubernetes. There is likely tons of software to further ease this process, so let's leave that open for other answers.

Answer 2

We're working on an open source project called Jenkins X which is a proposed sub project of the Jenkins foundation aimed at automating CI/CD on Kubernetes using Jenkins pipelines and GitOps for promotion across environments.

If you want to see how to automate CI/CD with multiple environments on Kubernetes using GitOps for promotion between environments and Preview Environments on Pull Requests you might wanna check out my recent talk on Jenkins X at DevOxx UK where I do a live demo of this on GKE.