Can I run Google Monitoring Agent inside a Kuberne

Question

It seems that the Google Monitoring Agent (powered by Stackdriver) should be installed on each Node (i.e. each compute instance, i.e. each machine) of a Kubernetes cluster.

However the new plugins, like Nginx, Redis, ElasticSearch..., need those agents to know the IP of these services. This means having kube-proxy running and set up which should mean running that Google Monitoring Agent on a Pod.

These two conflict: On one side that agent monitors the entire machine, on the other it monitor services running on one or more machines.

Can these Stackdriver plugins work on a Google Container Engine (GKE) / Kubernetes cluster?

Answer 1

I had tried to use a Stackdriver container in a pod to collect stats about Nginx/Uwsgi in the same pod. I had some findings that may be not so helpful. Just for your reference.

To create the stackdriver image, you may reference the docker file created by Keto. https://hub.docker.com/r/keto/stackdriver/~/dockerfile/

FROM centos:centos7

MAINTAINER Mikael Keto

# add stackdriver repository
RUN curl -o /etc/yum.repos.d/stackdriver.repo https://repo.stackdriver.com/stackdriver-el7.repo

# install stackdriver
RUN yum -y install initscripts stackdriver-agent && yum clean all

RUN mkdir -p /var/lock/subsys; exit 0
ADD run.sh /run.sh
RUN chmod 755 /run.sh

CMD ["/run.sh"]

The run.sh is look like below,

#!/usr/bin/env bash

/opt/stackdriver/stack-config --write-gcm --no-start
/etc/init.d/stackdriver-agent start

while true; do
    sleep 60
    agent_pid=$(cat /var/run/stackdriver-agent.pid 2>/dev/null)

    ps -p $agent_pid > /dev/null 2>&1
    if [ $? != 0 ]; then
        echo "Stackdriver agent pid not found!"
        break;
    fi
done

In the GKE/K8S deployment yaml file,

apiVersion: extensions/v1beta1
kind: Deployment
...
      - name: stackdriver-agent
        image: gcr.io/<project_id>/stackdriver-agent:<your_version>
        command: ['/run.sh']

In my test, I found

• It will report stats based on [node_name] instead of [container_name].
• It will collect many system stats that are meaningful for a node, but since it is in a pod, it's quite pointless.

Well, I hope to find some way to collect both statistics of the pods and nodes that I need, but I didn't find a easy way to do that. What I did is do that by Google Python API library, but that takes too much time.

Answer 2

You can install the StackDriver Agent in your Dockerfile.

I have been able to get this working for a couchdb container as follows:

FROM klaemo/couchdb

RUN apt-get update
RUN apt-get install curl lsb-release -y
RUN curl -O https://repo.stackdriver.com/stack-install.sh
RUN apt-get install libyajl2 -y

COPY couchdb.conf /opt/stackdriver/collectd/etc/collectd.d/couchdb.conf

CMD bash stack-install.sh --write-gcm && service stackdriver-agent restart && couchdb

Answer 3

There is an other way to use Dockerfile. When creating the docker image, pre-install necessary libraries for the stackdriver-agent installation.

FROM mongo
RUN apt-get update && apt-get install -y curl lsb-release

# COPY credential
COPY gcloud-credential.json /etc/google/auth/application_default_credentials.json
ENV GOOGLE_APPLICATION_CREDENTIALS "/etc/google/auth/application_default_credentials.json"

# download Stackdriver Agent installer
RUN curl -O https://repo.stackdriver.com/stack-install.sh
RUN chmod +x /stack-install.sh

# COPY stackdriver mongodb plugin
COPY mongodb.conf /opt/stackdriver/collectd/etc/collectd.d/mongodb.conf

Then install the agent using POD lifecycle.

spec:
  containers:
  - image: your_mongo_image
    name: my-mongo
    ports:
    - containerPort: 27017
    lifecycle:
      postStart:
        exec:
          command: ["/stack-install.sh", "--write-gcm"]

Answer 4

To monitor each machine (memory, CPU, disk...) it's possible to install the agent on each node (i.e. on each Compute Instance of your GKE cluster). Note that it'll not work with auto-scaling in the sense that re-created nodes won't have the agent installed.

To monitor services (number of requests/s, client connection...) it's possible to install the agent plugin in another container so that for example Nginx Pod run two containers:

• Nginx
• Google Monitoring Agent together with the Nginx plugin

Note: Not fully tested yet.