Logs in Kubernetes Pod not showing up

Question

I have Kubernetes set up and running a grpc service in a pod. I am successfully hitting an endpoint on the service, which has a print() statement in it, but I see no logs in the log file. I have seen this before when I was running a (cron) job in Kubernetes and the logs only appeared after the job was done (as opposed to when the job was running). Is there a way to make kubernetes write to the log file right away? Any setting that I can put (either cluster-level or just for the pod)? Thanks for any help in advance!

Answer 1

One possibility is that the container is starved for CPU. We have run into this issue when running locally on minikube with resource limits that enforced in our larger cluster. Try bumping the CPU resource limits on your pod. Below is an example yaml.

If your CPU limits are around 20-40m, that might be too low to run a full flask/python app. You might try bumping it to closer to 100m. It's not going to crush your local machine.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: python-app
spec:
  selector:
    matchLabels:
      app: python-app
  template:
    metadata:
      labels:
        app: python-app
    spec:
      containers:
      - name: python-app
        image: python-app
        imagePullPolicy: Never
        resources:
          requests:
            cpu: 40m
            memory: 40Mi
          limits:
            cpu: 20m
            memory: 20Mi

Answer 2

Here is an example of K8S deployment yaml so you can copy paste the solution from the aforementioned answer:

root@k8s:~/python_docker$ cat deployment.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hpa
spec:
  selector:
    matchLabels:
      app: hpa
  template:
    metadata:
      labels:
        app: hpa
    spec:
      containers:
      - name: hpa
        image: my-hpa/py
        env:
        - name: PYTHONUNBUFFERED
          value: "0"
        resources:
          requests:
            cpu: 100m
            memory: 200Mi

Answer 3

Found the root cause. Specifically, found it at Python app does not print anything when running detached in docker . The solution is to set the following environmental variable: PYTHONUNBUFFERED=0 . It was not that the print statement was not being displayed, it was that the print statement was being buffered. Doing the above will solve the issue.