Question

Please suggest the cause of the error of not being able to mount AWS EBS volume in pod.

journalctl -b -f -u kubelet

1480 kubelet.go:1625] Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx". list of unattached/unmounted volumes=[ebs]; skipping pod
1480 pod_workers.go:186] Error syncing pod ddc938ee-edda-11e7-ae06-06bb783bb15c ("nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)"), skipping: timeout expired waiting for volumes to attach/mount for pod "default"/"nginx". list of unattached/unmounted volumes=[ebs]
1480 reconciler.go:217] operationExecutor.VerifyControllerAttachedVolume started for volume "pv-ebs" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") pod "nginx" (UID: "ddc938ee-edda-11e7-ae06-06bb783bb15c")
1480 nestedpendingoperations.go:263] Operation for "\"kubernetes.io/aws-ebs/vol-0d275986ce24f4304\"" failed. No retries permitted until 2017-12-31 03:34:03.644604131 +0000 UTC m=+6842.543441523 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pv-ebs\" (UniqueName: \"kubernetes.io/aws-ebs/vol-0d275986ce24f4304\") pod \"nginx\" (UID: \"ddc938ee-edda-11e7-ae06-06bb783bb15c\") "

Steps

Deployed K8S 1.9 using kubeadm (without EBS volume mount, pods work) in AWS (us-west-1 and AZ is us-west-1b).
Configure an IAM role as per Kubernetes - Cloud Providers and kubelets failing to start when using 'aws' as cloud provider.
Assign the IAM role to EC2 instances as per Easily Replace or Attach an IAM Role to an Existing EC2 Instance by Using the EC2 Console.
Deploy PV/PVC/POD as in the manifest.

The status from the kubectl:

kubectl get

NAME      READY     STATUS              RESTARTS   AGE       IP        NODE
nginx     0/1       ContainerCreating   0          29m       <none>    ip-172-31-1-43.us-west-1.compute.internal

NAME        CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS    CLAIM             STORAGECLASS   REASON    AGE
pv/pv-ebs   5Gi        RWO            Recycle          Bound     default/pvc-ebs                            33m

NAME          STATUS    VOLUME    CAPACITY   ACCESS MODES   STORAGECLASS   AGE
pvc/pvc-ebs   Bound     pv-ebs    5Gi        RWO                           33m

kubectl describe pod nginx

Events:
  Type     Reason                 Age                From                                                Message
  ----     ------                 ----               ----                                                -------
  Normal   Scheduled              27m                default-scheduler                                   Successfully assigned nginx to ip-172-31-1-43.us-west-1.compute.internal
  Normal   SuccessfulMountVolume  27m                kubelet, ip-172-31-1-43.us-west-1.compute.internal  MountVolume.SetUp succeeded for volume "default-token-dt698"
  Warning  FailedMount            6s (x12 over 25m)  kubelet, ip-172-31-1-43.us-west-1.compute.internal  Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx".                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  Warning  FailedMount            6s (x12 over 25m)  kubelet, ip-172-31-1-43.us-west-1.compute.internal  Unable to mount volumes for pod "nginx_default(ddc938ee-edda-11e7-ae06-06bb783bb15c)": timeout expired waiting for volumes to attach/mount for pod "default"/"nginx".

Manifest

---
kind: PersistentVolume
apiVersion: v1
metadata:
  name: pv-ebs
  labels:
    type: amazonEBS
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-0d275986ce24f4304
    fsType: ext4
  persistentVolumeReclaimPolicy: Recycle
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: pvc-ebs
  labels:
    type: amazonEBS
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
kind: Pod
apiVersion: v1
metadata:
  name: nginx
spec:
  containers:
    - name: myfrontend
      image: nginx
      volumeMounts:
      - mountPath: "/var/www/html"
        name: ebs
  volumes:
    - name: ebs
      persistentVolumeClaim:
        claimName: pvc-ebs

IAM Policy

Environment

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T21:07:38Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T20:55:30Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

EC2

EBS

Solution

Found the documentation which shows how to configure AWS cloud provider.

K8S AWS Cloud Provider Notes

Steps

Tag EC2 instances and SG with the KubernetesCluster=${kubernetes cluster name}. If created with kubeadm, it is kubernetes as in Ability to configure user and cluster name in AdminKubeConfigFile
Run kubeadm init --config kubeadm.yaml.

kubeadm.yaml (Ansible template)

kind: MasterConfiguration
apiVersion: kubeadm.k8s.io/v1alpha1
api:
  advertiseAddress: {{ K8S_ADVERTISE_ADDRESS }}
networking:
  podSubnet:        {{ K8S_SERVICE_ADDRESSES }}
cloudProvider:      {{ K8S_CLOUD_PROVIDER }}

Result

$ journalctl -b -f CONTAINER_ID=$(docker ps | grep k8s_kube-controller-manager | awk '{ print $1 }')

Jan 02 04:48:28 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:28.752141
1 reconciler.go:287] attacherDetacher.AttachVolume started for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-3
Jan 02 04:48:39 ip-172-31-4-117.us-west-1.compute.internal dockerd-current[8063]: I0102 04:48:39.309178
1 operation_generator.go:308] AttachVolume.Attach succeeded for volume "kuard-pv" (UniqueName: "kubernetes.io/aws-ebs/vol-0d275986ce24f4304") from node "ip-172-

$ kubectl describe pod kuard
...
Volumes:
  kuard-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  kuard-pvc
    ReadOnly:   false

$ kubectl describe pv kuard-pv
Name:            kuard-pv
Labels:          failure-domain.beta.kubernetes.io/region=us-west-1
                 failure-domain.beta.kubernetes.io/zone=us-west-1b
                 type=amazonEBS
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"PersistentVolume","metadata":{"annotations":{},"labels":{"type":"amazonEBS"},"name":"kuard-pv","namespace":""},"spec":{"acce...
                 pv.kubernetes.io/bound-by-controller=yes
StorageClass:    
Status:          Bound
Claim:           default/kuard-pvc
Reclaim Policy:  Retain
Access Modes:    RWO
Capacity:        5Gi
Message:         
Source:
    Type:       AWSElasticBlockStore (a Persistent Disk resource in AWS)
    VolumeID:   vol-0d275986ce24f4304
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
Events:         <none>

$ kubectl version -o json
{
  "clientVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T21:07:38Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  },
  "serverVersion": {
    "major": "1",
    "minor": "9",
    "gitVersion": "v1.9.0",
    "gitCommit": "925c127ec6b946659ad0fd596fa959be43f0cc05",
    "gitTreeState": "clean",
    "buildDate": "2017-12-15T20:55:30Z",
    "goVersion": "go1.9.2",
    "compiler": "gc",
    "platform": "linux/amd64"
  }
}