Kubernetes 1.4 SSL Termination on AWS

I have 6 HTTP micro-services. Currently they run in a crazy bash/custom deploy tools setup (dokku, mup).

I dockerized them and moved to kubernetes on AWS (setup with kop). The last piece is converting my nginx config.

I'd like

All 6 to have SSL termination (not in the docker image)
4 need websockets and client IP session affinity (Meteor, Socket.io)
5 need http->https forwarding
1 serves the same content on http and https

I did 1. SSL termination setting the service type to LoadBalancer and using AWS specific annotations. This created AWS load balancers, but this seems like a dead end for the other requirements.

I looked at Ingress, but don't see how to do it on AWS. Will this Ingress Controller work on AWS?

Do I need an nginx controller in each pod? This looked interesting, but I'm not sure how recent/relevant it is.

I'm not sure what direction to start in. What will work?

Mike

回答1:

You should be able to use the nginx ingress controller to accomplish this.

SSL termination
Websocket support
http->https
Turn off the http->https redirect, as described in the link above

The README walks you through how to set it up, and there are plenty of examples.

The basic pieces you need to make this work are:

A default backend that will respond with 404 when there is no matching Ingress rule
The nginx ingress controller which will monitor your ingress rules and rewrite/reload nginx.conf whenever they change.
One or more ingress rules that describe how traffic should be routed to your services.

The end result is that you will have a single ELB that corresponds to your nginx ingress controller service, which in turn is responsible for routing to your individual services according to the ingress rules specified.

回答2:

There may be a better way to do this. I wrote this answer because I asked the question. It's the best I could come up with Pixel Elephant's doc links above.

The default-http-backend is very useful for debugging. +1

Ingress

this creates an endpoint on the node's IP address, which can change depending on where the Ingress Container is running
note the configmap at the bottom. Configured per environment.

(markdown placeholder because no ```)

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: "nginx"
  name: all-ingress
spec:
  tls:
  - hosts:
    - admin-stage.example.io
    secretName: tls-secret
  rules:
  - host: admin-stage.example.io
    http:
      paths:
      - backend:
          serviceName: admin
          servicePort: http-port
        path: /
---
apiVersion: v1
data:
  enable-sticky-sessions: "true"
  proxy-read-timeout: "7200"
  proxy-send-imeout: "7200"
kind: ConfigMap
metadata:
  name: nginx-load-balancer-conf

App Service and Deployment

the service port needs to be named, or you may get "upstream default-admin-80 does not have any active endpoints. Using default backend"

(markdown placeholder because no ```)

apiVersion: v1
kind: Service
metadata:
  name: admin
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  selector:
    app: admin
  sessionAffinity: ClientIP
  type: ClusterIP
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: admin
spec:
  replicas: 1
  template:
    metadata:
      labels: 
        app: admin
      name: admin
    spec:
      containers:
      - image: example/admin:latest
        name: admin
        ports:
        - containerPort: 80
          name: http-port
        resources:
          requests:
            cpu: 500m
            memory: 1000Mi
        volumeMounts:
        - mountPath: /etc/env-volume
          name: config
          readOnly: true
      imagePullSecrets:
      - name: cloud.docker.com-pull
      volumes:
      - name: config
        secret:
          defaultMode: 420
          items:
          - key: admin.sh
            mode: 256
            path: env.sh
          - key: settings.json
            mode: 256
            path: settings.json
          secretName: env-secret

Ingress Nginx Docker Image

note default-ssl-certificate at bottom
logging is great -v below
note the Service will create an ELB on AWS which can be used to configure DNS.

(markdown placeholder because no ```)

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress-service
spec:
  ports:
  - name: http-port
    port: 80
    protocol: TCP
    targetPort: http-port
  - name: https-port
    port: 443
    protocol: TCP
    targetPort: https-port
  selector:
    app: nginx-ingress-service
  sessionAffinity: None
  type: LoadBalancer
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx-ingress-controller
  labels:
    k8s-app: nginx-ingress-lb
spec:
  replicas: 1
  selector:
    k8s-app: nginx-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-lb
        name: nginx-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - image: gcr.io/google_containers/nginx-ingress-controller:0.8.3
        name: nginx-ingress-lb
        imagePullPolicy: Always
        readinessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
        livenessProbe:
          httpGet:
            path: /healthz
            port: 10254
            scheme: HTTP
          initialDelaySeconds: 10
          timeoutSeconds: 1
        # use downward API
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace
        ports:
        - name: http-port
          containerPort: 80
          hostPort: 80
        - name: https-port
          containerPort: 443
          hostPort: 443
        # we expose 18080 to access nginx stats in url /nginx-status
        # this is optional
        - containerPort: 18080
          hostPort: 18080
        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --default-ssl-certificate=default/tls-secret
        - --nginx-configmap=$(POD_NAMESPACE)/nginx-load-balancer-conf
        - --v=2

Default Backend (this is copy/paste from .yaml file)

apiVersion: v1
kind: Service
metadata:
  name: default-http-backend
  labels:
    k8s-app: default-http-backend
spec:
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
    name: http
  selector:
    k8s-app: default-http-backend
---
apiVersion: v1
kind: ReplicationController
metadata:
  name: default-http-backend
spec:
  replicas: 1
  selector:
    k8s-app: default-http-backend
  template:
    metadata:
      labels:
        k8s-app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

This config uses three secrets:

tls-secret - 3 files: tls.key, tls.crt, dhparam.pem
env-secret - 2 files: admin.sh and settings.json. Container has start script to setup environment.
cloud.docker.com-pull