Google Container Engine stdout Logs Not Showing Up

Question

My stdout logs are not showing up in Google Logs Viewer, or when using kubectl logs <pod>. The cluster has Cloud Logging enabled and fluentd containers are running on each node.

Example Python code:

logger = logging.getLogger()
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.INFO)
logger.addHandler(handler)
logger.info("test log")

The "counter-pod" example from their docs does work on my cluster, so the fluentd containers are picking up stdout and sending it to Logs Viewer.

Any suggestions for things I should try? Thanks in advance.

Answer 1

The logs are definitely going to stdout, they just aren't showing up when running kubectl logs <pod_name>. Nor are they showing up in Google Logs Viewer.

This is because logs sent to stdout will only be captured if they're coming from the process that's the entry point of the Docker container. Things that are done in the shell or via a cron job don't show up.

In my case I had a cron job that was invoking a script. By running the script as the container's entry point, the logs showed up fine.

Answer 2

If the logs aren't showing up when you run kubectl logs pod, then the problem is almost certainly that the logs from your application aren't going to stdout or stderr, so that's what you should focus on.

What happens if you docker run your application locally? Does docker logs then show the logs you expect? That should at least get you a faster iteration cycle to play around with different approaches to fixing it.