best way in Kubernetes to receive notifications of

Question

I am using fabric8 to develop a cluster management layer on top of Kubernetes, and I am confused as to what the 'official' API is to obtain notifications of errors when things go wrong when instantiating pods/rep controllers & services etc.

In the section "Pod Deployment Code" I have a stripped down version of what we do for pods. In the event that everything goes correctly, our code is fine. We rely on setting 'watches' as you can see in the method deployPodWithWatch. All I do in the given eventReceived callback is to print the event, but our real code will break apart a notification like this:

got action: 
    MODIFIED / 
      Pod(apiVersion=v1, kind=Pod, metadata=...etc etc
        status=PodStatus(
            conditions=[

and pick out the 'status' element of the Pod and when we get PodCondition(status=True, type=Ready), we know that our pod has been successfully deployed.

In the happy path case this works great. And you can actually run the code supplied with variable k8sUrl set to the proper url for your site (hopefully your k8s installation does not require auth which is site specific so i didn't provide code for that).

However, suppose you change the variable imageName to "nginBoo". There is no public docker image of that name, so after you run the code, set your kubernetes context to the namespace "junk", and do a

  describe pod podboy

you will see two status messages at the end with the following values for Reason / Message

Reason      message
failedSync  Error syncing pod, skipping...
failed      Failed to pull image "nginBoo": API error (500): 
            Error parsing reference: "nginBoo" 
            is not a valid repository/tag

I would like to implement a watch callback so that it catches these types of errors. However, the only thing that I see are 'MODIFIED' events wherein the Pod has a field like this:

 state=ContainerState(running=null, terminated=null, 
        waiting=ContainerStateWaiting(
            reason=API error (500): 
                Error parsing reference: 
                    "nginBoo" is not a valid repository/tag

I suppose I could look for a reason code that contained the string 'API error' but this seems to be very much an implementation-dependent hack -- it might not cover all cases, and maybe it will change under my feet with future versions. I'd like some more 'official' way of figuring out if there was an error, but my searches have come up dry -- so I humbly request guidance from all of you k8s experts out there. Thanks !

Pod Deployment Code

import com.fasterxml.jackson.databind.ObjectMapper
import scala.collection.JavaConverters._
import com.ning.http.client.ws.WebSocket
import com.typesafe.scalalogging.StrictLogging
import io.fabric8.kubernetes.api.model.{DoneableNamespace, Namespace, Pod, ReplicationController}
import io.fabric8.kubernetes.client.DefaultKubernetesClient.ConfigBuilder
import io.fabric8.kubernetes.client.Watcher.Action
import io.fabric8.kubernetes.client.dsl.Resource
import io.fabric8.kubernetes.client.{DefaultKubernetesClient, Watcher}

object ErrorTest extends App with StrictLogging {
  // corresponds to --insecure-skip-tls-verify=true, according to io.fabric8.kubernetes.api.model.Cluster
  val trustCerts = true
  val k8sUrl = "http://localhost:8080"
  val namespaceName = "junk" // replace this with name of a namespace that you know exists
  val imageName: String = "nginx"

  def go(): Unit = {
    val kube = getConnection
    dumpNamespaces(kube)
    deployPodWithWatch(kube, getPod(image = imageName))
  }

  def deployPodWithWatch(kube: DefaultKubernetesClient, pod: Pod): Unit = {
    kube.pods().inNamespace(namespaceName).create(pod) /*  create the pod ! */
    val podWatchWebSocket: WebSocket =                /* create watch on the pod */
      kube.pods().inNamespace(namespaceName).withName(pod.getMetadata.getName).watch(getPodWatch)
  }

  def getPod(image: String): Pod = {
    val jsonTemplate =
      """
    |{
    | "kind": "Pod",
    | "apiVersion": "v1",
    | "metadata": {
    |   "name": "podboy",
    |   "labels": {
    |     "app": "nginx"
    |   }
    | },
    | "spec": {
    |   "containers": [
    |     {
    |     "name": "podboy",
    |     "image": "<image>",
    |     "ports": [
    |       {
    |         "containerPort": 80,
    |         "protocol": "TCP"
    |       }
    |     ]
    |     }
    |   ]
    | }
    |}
      """.
    stripMargin
    val replacement: String = "image\": \"" + image
    val json = jsonTemplate.replaceAll("image\": \"<image>", replacement)
    System.out.println("json:" + json);
    new ObjectMapper().readValue(json, classOf[Pod])
  }

  def dumpNamespaces(kube: DefaultKubernetesClient): Unit = {
    val namespaceNames = kube.namespaces().list().getItems.asScala.map {
      (ns: Namespace) => {
    ns.getMetadata.getName
      }
    }
    System.out.println("namespaces are:" + namespaceNames);
  }

  def getConnection = {
    val configBuilder = new ConfigBuilder()
    val config =
      configBuilder.
    trustCerts(trustCerts).
    masterUrl(k8sUrl).
    build()
    new DefaultKubernetesClient(config)
  }

  def getPodWatch: Watcher[Pod] = {
    new Watcher[Pod]() {
      def eventReceived(action: Action, watchedPod: Pod) {
       System.out.println("got action: " + action + " / "  + watchedPod)
      }
    }
  }

  go()
}

Answer 1

I'd suggest you to have a look at events, see this topic for some guidance. Generally each object should generate events you can watch and be notified of such errors.