Forbidden: updates to statefulset spec for fields

Question

I use io.fabric8.kubernetes-client, version 3.1.8 to do RollingUpdate of kubernetes resource. It is fine for Deployment. But I meet an exception for StatefulSet. But it is also fine if I use 'kubectl apply -f ***.yaml' for the StatefulSet.

Code to RollingUpdate Deployment:

public void createOrReplaceResourceByYaml(String namespace, KubernetesResource resource) {
  KubernetesClient client = k8sRestClient.newKubeClient();
  Deployment deployment = (Deployment) resource;
  logger.info(String.format("Create/Replace Deployment [%s] in namespace [%s].", ((Deployment) resource).getMetadata().getName(), namespace));
  NonNamespaceOperation<Deployment, DeploymentList, DoneableDeployment, ScalableResource<Deployment, DoneableDeployment>> deployments = client.extensions().deployments().inNamespace(namespace);
  Deployment result = deployments.createOrReplace(deployment);
  logger.info(String.format("Created/Replaced Deployment [%s].", result.getMetadata().getName()));
}

Code to RollingUpdate StatefulSet

public void createOrReplaceResourceByYaml(String namespace, KubernetesResource resource) {
  KubernetesClient client = k8sRestClient.newKubeClient();
  StatefulSet statefulSet = (StatefulSet) resource;
  logger.info(String.format("Create/Replace StatefulSet [%s] in namespace [%s].", statefulSet.getMetadata().getName(), namespace));
  NonNamespaceOperation<StatefulSet, StatefulSetList, DoneableStatefulSet, RollableScalableResource<StatefulSet, DoneableStatefulSet>> statefulSets = client.apps().statefulSets().inNamespace(namespace);
  StatefulSet result = statefulSets.createOrReplace(statefulSet);
  logger.info(String.format("Created/Replaced StatefulSet [%s].", result.getMetadata().getName()));
}

Exception when do RollingUpdate of StatefulSet

Failure executing: PUT at: https://kubernetes.default.svc/apis/apps/v1beta1/namespaces/itsma1/statefulsets/pro-rabbitmq. Message: StatefulSet.apps "pro-rabbitmq" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden.. Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec, message=Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden., reason=FieldValueForbidden, additionalProperties={})], group=apps, kind=StatefulSet, name=pro-rabbitmq, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=StatefulSet.apps "pro-rabbitmq" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden., metadata=ListMeta(resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).

I am curious why the error happened and how to fix it.

Answer 1

You can try this to update the StatefulSet

client.apps().statefulSets().withName("repl1").rolling().withTimeout(5, TimeUnit.MINUTES).updateImage("");

If you want to only scale, you can try this

client.apps().statefulSets().withName("repl1").scale(5, true);

Answer 2

I had this problem recently too, and I found the problem is the client tries to modify spec->selector->matchLabels->deployment, then the server throw that error back since that field is not editable based on the error message. So, I filed an issue to them.

However, if you want true "rolling" update of your stateful set and your kube cluster is recent enough, you could try to do

k8client.apps().statefulSets().inNamespace(namespace).withName(name).cascading(false).replace(statefulSet)

The cascading(false) seems did the trick, it basically tells the client just update the stateful set without scaling down the pods first. And cluster will handle the rolling process for you if your update strategy is rolling.

Answer 3

In StatefulSet, unlike Deployment, you can update only limited number of values - replicas, template, and updateStrategy.

You issue happening because Fabric trying to update values which is impossible to update.

The only thing you can do is carefully prepare a new statefulSet object which will have a same name as old but contain only values which you can update.

Alternative way is to delete old statefulSet before upload a new one with a same name.

Also, try to use a Kubernetes version upper 1.9 if you don't, because statefulSet is officially stable only in 1.9 and above.

BTW, here is a bug in Fabric's GitHub which can effect your code.