Incorrect reporting of container memory usage by c

Question

cAdvisor reports 14GB for the memory used by Prometheus where as top reports 6.xGB.

Can someone explain why this discrepancy?

The documentatation of container_memory_usage_bytes says

Current memory usage in bytes, including all memory regardless of when it was accessed

but it's not clear what this refers to - I assume it's virtual memory size?

As reported by cAdvisor:

core@ip-172-20-100-148 ~ $ curl -q localhost:4194/metrics | grep container_memory_usage_bytes | grep prometheus
container_memory_usage_bytes{container_name="prometheus",id="/docker/d37e7503309e632265cb834095efa949da4dc3c72122cb290e626f1121d0ed6b",image="quay.io/prometheus/prometheus:v1.7.1",name="k8s_prometheus.71f5f8c6_prometheus-zwvhx_xx-system_9753de21-aaac-11e7-97e5-026a05f72f78_4682ea0b",namespace="xx-system",pod_name="prometheus-zwvhx"} 1.4178545664e+10

As reported by top on the host machine:

Tasks: 272 total,   1 running, 271 sleeping,   0 stopped,   0 zombie
%Cpu(s): 20.2 us,  2.7 sy,  0.0 ni, 75.9 id,  0.5 wa,  0.6 hi,  0.1 si,  0.0 st
KiB Mem:  65964640 total, 64892592 used,  1072048 free,  5222520 buffers
KiB Swap:        0 total,        0 used,        0 free.  4737420 cached Mem

   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                   
118446 root      20   0 6783108 6.154g  19864 S 316.7  9.8   8241:05 prometheus 

Answer 1

Per the discussion here, cAdvisor reports a total memory usage in bytes.

In your case, cAdvisor was reporting 1.4178545664e+10 Byte = 14.178545664 Gigabyte. Note it was e+10 so it is approximately 14G, not 1.4G.