“Memory used” metric: Go tool pprof vs docker stat

I wrote a golang application running in each of my docker containers. It communicates with each other using protobufs via tcp and udp and I use Hashicorp's memberlist library to discover each of the containers in my network. On docker stats I see that the memory usage is linearly increasing so I am trying to find any leaks in my application.

Since it is an application which keeps running, am using http pprof to check the live application in any one of the containers. I see that runtime.MemStats.sys is constant even though docker stats is linearly increasing. My --inuse_space is around 1MB and --alloc_space ofcourse keeps increasing over time. Here is a sample of alloc_space:

root@n3:/app# go tool pprof --alloc_space main http://localhost:8080/debug/pprof/heap                                                                                                                       
Fetching profile from http://localhost:8080/debug/pprof/heap
Saved profile in /root/pprof/pprof.main.localhost:8080.alloc_objects.alloc_space.005.pb.gz
Entering interactive mode (type "help" for commands)
(pprof) top --cum
1024.11kB of 10298.19kB total ( 9.94%)
Dropped 8 nodes (cum <= 51.49kB)
Showing top 10 nodes out of 34 (cum >= 1536.07kB)
      flat  flat%   sum%        cum   cum%
         0     0%     0% 10298.19kB   100%  runtime.goexit
         0     0%     0%  6144.48kB 59.67%  main.Listener
         0     0%     0%  3072.20kB 29.83%  github.com/golang/protobuf/proto.Unmarshal
  512.10kB  4.97%  4.97%  3072.20kB 29.83%  github.com/golang/protobuf/proto.UnmarshalMerge
         0     0%  4.97%  2560.17kB 24.86%  github.com/hashicorp/memberlist.(*Memberlist).triggerFunc
         0     0%  4.97%  2560.10kB 24.86%  github.com/golang/protobuf/proto.(*Buffer).Unmarshal
         0     0%  4.97%  2560.10kB 24.86%  github.com/golang/protobuf/proto.(*Buffer).dec_struct_message
         0     0%  4.97%  2560.10kB 24.86%  github.com/golang/protobuf/proto.(*Buffer).unmarshalType
  512.01kB  4.97%  9.94%  2048.23kB 19.89%  main.SaveAsFile
         0     0%  9.94%  1536.07kB 14.92%  reflect.New
(pprof) list main.Listener
Total: 10.06MB
ROUTINE ======================== main.Listener in /app/listener.go
         0        6MB (flat, cum) 59.67% of Total
         .          .     24:   l.SetReadBuffer(MaxDatagramSize)
         .          .     25:   defer l.Close()
         .          .     26:   m := new(NewMsg)
         .          .     27:   b := make([]byte, MaxDatagramSize)
         .          .     28:   for {
         .   512.02kB     29:       n, src, err := l.ReadFromUDP(b)
         .          .     30:       if err != nil {
         .          .     31:           log.Fatal("ReadFromUDP failed:", err)
         .          .     32:       }
         .   512.02kB     33:       log.Println(n, "bytes read from", src)
         .          .     34:       //TODO remove later. For testing Fetcher only
         .          .     35:       if rand.Intn(100) < MCastDropPercent {
         .          .     36:           continue
         .          .     37:       }
         .        3MB     38:       err = proto.Unmarshal(b[:n], m)
         .          .     39:       if err != nil {
         .          .     40:           log.Fatal("protobuf Unmarshal failed", err)
         .          .     41:       }
         .          .     42:       id := m.GetHead().GetMsgId()
         .          .     43:       log.Println("CONFIG-UPDATE-RECEIVED { \"update_id\" =", id, "}")
         .          .     44:       //TODO check whether value already exists in store?
         .          .     45:       store.Add(id)
         .        2MB     46:       SaveAsFile(id, b[:n], StoreDir)
         .          .     47:       m.Reset()
         .          .     48:   }
         .          .     49:}
(pprof)

I have been able to verify that no goroutine leak is happening using http://:8080/debug/pprof/goroutine?debug=1

Please comment on why docker stats shows a different picture (linearly increasing memory)

CONTAINER           CPU %               MEM USAGE / LIMIT       MEM %               NET I/O               BLOCK I/O           PIDS
n3                  0.13%               19.73 MiB / 31.36 GiB   0.06%               595 kB / 806 B        0 B / 73.73 kB      14

If I run it over night, this memory bloats to around 250MB. I have not run it longer than that, but I feel this should have reached a plateau instead of increasing linearly

docker stats shows the memory usage stats from cgroups. (Refer: https://docs.docker.com/engine/admin/runmetrics/)

If you read the "outdated but useful" documentation (https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt) it says

5.5 usage_in_bytes

For efficiency, as other kernel components, memory cgroup uses some optimization to avoid unnecessary cacheline false sharing. usage_in_bytes is affected by the method and doesn't show 'exact' value of memory (and swap) usage, it's a fuzz value for efficient access. (Of course, when necessary, it's synchronized.) If you want to know more exact memory usage, you should use RSS+CACHE(+SWAP) value in memory.stat(see 5.2).

Page Cache and RES are included in the memory usage_in_bytes number. So if the container has File I/O, the memory usage stat will increase. However, for a container, if the usage hits that maximum limit, it reclaims some of the memory which is unused. Hence, when I added a memory limit to my container, I could observe that the memory is reclaimed and used when the limit is hit. The container processes are not killed unless there is no memory to reclaim and a OOM error happens. For anyone concerned with the numbers shown in docker stats, the easy way is to check the detailed stats available in cgroups at the path: /sys/fs/cgroup/memory/docker// This shows all the memory metrics in detail in memory.stats or other memory.* files.

If you want to limit the resources used by the docker container in the "docker run" command you can do so by following this reference: https://docs.docker.com/engine/admin/resource_constraints/

Since I am using docker-compose, I did it by adding a line in my docker-compose.yml file under the service I wanted to limit: