Kubernetes Endpoints with TTL

Question

I have a Kubernetes service without a selector for which I would like to manually manage the Endpoints by having the endpoint servers register/heartbeat themselves.

Is there a way to specify a TTL for Endpoints when I POST them to the Kubernetes API server, so that they will timeout and be deleted automatically if my endpoint server terminates and stops heartbeating?

If not, would it be reasonable if I add the Endpoints to the registry by POSTing directly to the underlying Etcd, instead of going through the Kubernetes API, or will that cause other problems?

Answer 1

You do not need to modify kubernetes to do this.

Here is how to do it yourself.

1. add an annotation to each object that you want to have a TTL. The annotation can say when it should expire. You can pick the name and format of this annotation.
2. update the annotation each time you update the object.
3. run another process that repeatedly lists all the objects of a given type and deletes ones that need to expire.

Here are specific commands to do this for endpoints.

Add an annotation to an endpoint with expiration time one minute from now:

   #!/bin/bash
   expiretime=$(date -v+60S +%s)
   kubectl annotate endpoints/somename expires-at=$expiretime

Script to list endpoints, and delete those with expires-at after now:

   #!/bin/bash
   while 1
   do 
     for NS in $(kubectl get namespaces -o name | cut -f 2 -d "/")
     do 
         for NAME in $(kubectl --namespace=$NS get endpoints -o name)
         do
             exp=$( kubectl get --namespace $NS $NAME -o jsonpath={.metadata.annotations."expires-at"} 2> /dev/null) && \
             [[ $exp < $(date +%s) ]] && \
             echo "Deleting expired endpoints $NAME in $NS" && \ 
             kubectl delete $NS $NAME
         done
     done
   done

A pod is a great place to run the above script. It will have automatic access to the API and with a replication controller, it will run forever.

Answer 2

There is no TTL or heartbeat built into the endpoints API objects. You really don't want to write directly to etcd though… that will bite you eventually