I'm seeing a strange problem in a Docker/Fig environment. My hypothesis is that it's due to a delay in mounting volumes to containers, but I'm not sure how to confirm that.
I have a container with the following:
Dockerfile
FROM busybox
MAINTAINER Dan Rumney <email>
ADD loadsnapshot.sh /loadsnapshot.sh
RUN ["chmod", "u+x", "/loadsnapshot.sh"]
VOLUME ["/snapshot"]
ENTRYPOINT ["/loadsnapshot.sh"]
loadsnapshot.sh
#!/bin/sh
if [ "$( ls -A /snapshot)" ]; then
echo "Loading snapshot..."
# Do stuff
else
echo "No snapshot to load"
fi
In my fig.yml file I have:
pdsvol:
image: busybox
volumes:
- "/opt/alfresco/alf_data"
- "/data"
- "/mysqlbackup"
- "/ldapbackup"
loader:
image: "docker.myregistry.com/snapshot.loader:3.5.0"
volumes_from:
- pdsvol
volumes:
- "/opt/snapshots/my-data/:/snapshot/"
The goal here (which may be obvious) is to start up a data container (pdsvol
) and then populate it with some data that's running on my machine. pdsvol
is then shared by a bunch of other containers.
The way I run this is to call
fig up pdsvol
and then
fig run --rm loader
What I expect to see is
builder@beast:/opt/docker-vm$ fig run --rm loader
Loading snapshot...
... stuff ...
Removing dockervm_loader_run_1...
and, sometimes I do. However, sometimes I see:
builder@beast:/opt/docker-vm$ fig run --rm loader
No snapshot to load
Removing dockervm_loader_run_1...
I can run fig run --rm loader
over and over and I will get one of the two outcomes.
My working theory was that there was some delay in mounting the volume and sometimes it happens before the ENTRYPOINT
script is run and sometimes after. However, if I run:
docker run --rm -v /opt/snapshots/my-data/:/snapshot/ busybox ls -A /snapshot
I consistently see the files I'm expecting... so this goes against that theory.
I know I could hack at loadsnapshot.sh
and put in a delay and see if that helps, but I'd rather understand what's going on than kludge a fix.
Does anyone have any ideas what's going on here?
BTW: the host system is Linux, so we are using native containers here.
Update
I tried putting a 2s delay at the top of loadsnapshot.sh
, but it did not help.
Update 2
I added some logging to fig to dump the configuration that is used to create the container and in every instance (fail or no), it's the same:
{
'Env': None,
'Hostname': None,
'Entrypoint': None,
'Dns': None,
'Memory': 0,
'OpenStdin': True,
'User': None,
'CpuShares': None,
'AttachStdout': True,
'NetworkDisabled': False,
'WorkingDir': None,
'Cmd': None,
'StdinOnce': True,
'AttachStdin': True,
'Volumes': {u'/snapshot/': {}},
'MemorySwap': 0,
'VolumesFrom': None,
'Tty': True,
'AttachStderr': True,
'Domainname': None,
'Image': 'docker.myregistry.com/snapshot.loader:3.5.0',
'ExposedPorts': None
}