EKS - Node labels

Question

Is there a way to add node labels when deploying worker nodes in EKS. I do not see an option in the CF template available for worker nodes.

EKS-CF-Workers

The only option I see right now is to use kubectl label command to add labels which is post cluster setup. However, the need to have complete automation which means applications are deployed automatically post cluster deployments and labels help in achieving the segregation.

Answer 1

You'll need to add the config in user_data and use the --node-labels option for the kubelet. Here's an example user_data which includes node_labels:

NodeLaunchConfig:
Type: AWS::AutoScaling::LaunchConfiguration
Properties:
  AssociatePublicIpAddress: 'true'
  IamInstanceProfile: !Ref NodeInstanceProfile
  ImageId: !Ref NodeImageId
  InstanceType: !Ref NodeInstanceType
  KeyName: !Ref KeyName
  SecurityGroups:
  - !Ref NodeSecurityGroup
  UserData:
    Fn::Base64:
      Fn::Join: [
        "",
        [
          "#!/bin/bash -xe\n",
          "CA_CERTIFICATE_DIRECTORY=/etc/kubernetes/pki", "\n",
          "CA_CERTIFICATE_FILE_PATH=$CA_CERTIFICATE_DIRECTORY/ca.crt", "\n",
          "MODEL_DIRECTORY_PATH=~/.aws/eks", "\n",
          "MODEL_FILE_PATH=$MODEL_DIRECTORY_PATH/eks-2017-11-01.normal.json", "\n",
          "mkdir -p $CA_CERTIFICATE_DIRECTORY", "\n",
          "mkdir -p $MODEL_DIRECTORY_PATH", "\n",
          "curl -o $MODEL_FILE_PATH https://s3-us-west-2.amazonaws.com/amazon-eks/1.10.3/2018-06-05/eks-2017-11-01.normal.json", "\n",
          "aws configure add-model --service-model file://$MODEL_FILE_PATH --service-name eks", "\n",
          "aws eks describe-cluster --region=", { Ref: "AWS::Region" }," --name=", { Ref: ClusterName }," --query 'cluster.{certificateAuthorityData: certificateAuthority.data, endpoint: endpoint}' > /tmp/describe_cluster_result.json", "\n",
          "cat /tmp/describe_cluster_result.json | grep certificateAuthorityData | awk '{print $2}' | sed 's/[,\"]//g' | base64 -d >  $CA_CERTIFICATE_FILE_PATH", "\n",
          "MASTER_ENDPOINT=$(cat /tmp/describe_cluster_result.json | grep endpoint | awk '{print $2}' | sed 's/[,\"]//g')", "\n",
          "INTERNAL_IP=$(curl -s http://169.254.169.254/latest/meta-data/local-ipv4)", "\n",
          "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /var/lib/kubelet/kubeconfig", "\n",
          "sed -i s,CLUSTER_NAME,", { Ref: ClusterName }, ",g /var/lib/kubelet/kubeconfig", "\n",
          "sed -i s,REGION,", { Ref: "AWS::Region" }, ",g /etc/systemd/system/kubelet.service", "\n",
          "sed -i s,MAX_PODS,", { "Fn::FindInMap": [ MaxPodsPerNode, { Ref: NodeInstanceType }, MaxPods ] }, ",g /etc/systemd/system/kubelet.service", "\n",
          "sed -i s,MASTER_ENDPOINT,$MASTER_ENDPOINT,g /etc/systemd/system/kubelet.service", "\n",
          "sed -i s,INTERNAL_IP,$INTERNAL_IP,g /etc/systemd/system/kubelet.service", "\n",
          "DNS_CLUSTER_IP=10.100.0.10", "\n",
          "if [[ $INTERNAL_IP == 10.* ]] ; then DNS_CLUSTER_IP=172.20.0.10; fi", "\n",
          "sed -i s,DNS_CLUSTER_IP,$DNS_CLUSTER_IP,g  /etc/systemd/system/kubelet.service", "\n",
          "sed -i s,CERTIFICATE_AUTHORITY_FILE,$CA_CERTIFICATE_FILE_PATH,g /var/lib/kubelet/kubeconfig" , "\n",
          "sed -i s,CLIENT_CA_FILE,$CA_CERTIFICATE_FILE_PATH,g  /etc/systemd/system/kubelet.service" , "\n"
          "sed -i s,INTERNAL_IP/a,--node-labels tier=development,g  /etc/systemd/system/kubelet.service" , "\n"
          "systemctl daemon-reload", "\n",
          "systemctl restart kubelet", "\n",
          "/opt/aws/bin/cfn-signal -e $? ",
          "         --stack ", { Ref: "AWS::StackName" },
          "         --resource NodeGroup ",
          "         --region ", { Ref: "AWS::Region" }, "\n"
        ]
      ]

The relevant line is:

"sed -i s,INTERNAL_IP/a,--node-labels tier=development,g  /etc/systemd/system/kubelet.service" , "\n"

WARNING: I haven't tested this, but I do something similar and it works fine

Answer 2

With the new EKS-optimized AMIs(amazon-eks-node-vXX) and Cloudformation template refactors provided by AWS it is now possible to add node labels as simple as providing arguments to the BootstrapArguments parameter of the [amazon-eks-nodegroup.yaml][1] Cloudfomation template. For example --kubelet-extra-args --node-labels=my-key=my-value. For more details check the AWS announcement: Improvements for Amazon EKS Worker Node Provisioning

Answer 3

I've managed to get it work with the next sed expression:

sed -i '/--node-ip/ a \ \ --node-labels group=node \\' /etc/systemd/system/kubelet.service