I have a service which I've configured to have 3 replicas (in a swarm of 3 nodes). When it comes time to update the service to a new version, I run this:
docker stack deploy -c docker-compose.yml authentication
I've also tried to force an update using:
docker service update --force authentication_authentication
One instance will get updated, the others will remain on the same version. The update command above hangs here forever:
overall progress: 1 out of 3 tasks
1/3: running [==================================================>]
2/3:
3/3:
In the systemd logs for docker, I see these entries:
Sep 10 10:24:28 docker01 dockerd[1478]: time="2018-09-10T10:24:28.910584651-04:00" level=warning msg="rmServiceBinding 489e7143d27eaa4e4210b080f77b60018a033b29a291599d63def87fb0bed903 possible transient state ok:false entries:0 set:false "
Sep 10 10:24:28 docker01 dockerd[1478]: time="2018-09-10T10:24:28.911078594-04:00" level=warning msg="rmServiceBinding fa961a79dc139cfcc31e7ae671b6305f83887d804c4ce28aaae6f7221de08021 possible transient state ok:false entries:0 set:false "
How can I get these services to update?
Additional details:
- Docker 18.06.0-ce
- Ubuntu 16.04.5 LTS
docker-compose.yml:
version: "3.1"
services:
authentication:
image: "<private registry>/authentication:1.0.11"
ports:
- 8008:8008
deploy:
mode: replicated
replicas: 3
placement:
constraints:
- node.platform.os==linux
Turned debug on in the docker config file and am now seeing this repeatedly in the log:
Sep 10 13:14:08 docker01 dockerd[1532]: time="2018-09-10T13:14:08.283136216-04:00" level=debug msg="Calling GET /v1.38/nodes"
Sep 10 13:14:08 docker01 dockerd[1532]: time="2018-09-10T13:14:08.487369825-04:00" level=debug msg="Calling GET /v1.38/services/authentication_authentication?insertDefaults=false"
Sep 10 13:14:08 docker01 dockerd[1532]: time="2018-09-10T13:14:08.488713081-04:00" level=debug msg="error handling rpc" error="rpc error: code = NotFound desc = service authentication_authentication not found" rpc=/docker.swarmkit.v1.Control/GetService
Sep 10 13:14:08 docker01 dockerd[1532]: time="2018-09-10T13:14:08.492766895-04:00" level=debug msg="Calling GET /v1.38/tasks?filters=%7B%22_up-to-date%22%3A%7B%22true%22%3Atrue%7D%2C%22service%22%3A%7B%22authentication_authentication%22%3Atrue%7D%7D"
Sep 10 13:14:08 docker01 dockerd[1532]: time="2018-09-10T13:14:08.494021275-04:00" level=debug msg="error handling rpc" error="rpc error: code = NotFound desc = service authentication_authentication not found" rpc=/docker.swarmkit.v1.Control/GetService
