How to know the reason why a docker container exit

Question

I have a Docker container running in a host of 1G RAM (there are also other containers running in the same host). The application in this Docker container will decode some images, which may consume memory a lot.

From time to time, this container will exit. I doubt it is due to out of memory but not very sure. I need a method to find the root cause. So is there any way to know what happened for this container's death?

Answer 1

Others have mentioned docker logs $container_id to view the output of the application. This would always be my first thing to check.

Next, you can run a docker inspect $container_id to view details on the state, e.g.:

    "State": {
        "Status": "exited",
        "Running": false,
        "Paused": false,
        "Restarting": false,
        "OOMKilled": false,
        "Dead": false,
        "Pid": 0,
        "ExitCode": 2,
        "Error": "",
        "StartedAt": "2016-06-28T21:26:53.477229071Z",
        "FinishedAt": "2016-06-28T21:26:53.478066987Z"
    },

The important line there is "OOMKilled" which will be true if you exceed the container memory limits and Docker kills your app. You may also want to lookup the exit code to see if it identifies a cause for the exit by your app.

Answer 2

You can find out whether the process inside the container was OOMkilled by reading the logs. OOMkills are initiated by the kernel so every time it happens there's a bunch of lines in /var/log/kern.log, for example:

python invoked oom-killer: gfp_mask=0x14000c0(GFP_KERNEL), nodemask=(null), order=0, oom_score_adj=995
oom_kill_process+0x22e/0x450
Memory cgroup out of memory: Kill process 31204 (python) score 1994 or sacrifice child
Killed process 31204 (python) total-vm:7350860kB, anon-rss:4182920kB, file-rss:2356kB, shmem-rss:0kB