My DockerFile contains the following instruction:
rm -f plugins.7z
This command worked as expected in earlier versions of docker but fails with version 1.13. I see the error:
cannot access plugins.7z: No such file or directory
If I bring up a container with the base image and execute the command manually, I see the same error.
Trying to list the folder contents displays:
# ls -lrt
ls: cannot access plugins.7z: No such file or directory
total 12
??????????? ? ? ? ? ? plugins.7z
This is not listed as a known issue in Docker Issues. How do I debug the issue further?
Edit:
- For reasons of IP, I cannot post the full Dockerfile here. Also, it may not be necessary. As I mentioned, I am able to simulate the issue even by manually running the container and trying to execute the command
- The file exists before I attempt to delete it
- I was wrong about there not being a similar bug in the issues list. Here is one
- The issue may not be to do with that file. Deleting other files/folders in the folder also makes them appear with ??? permissions
- The user performing the operation is root
The reason removing directories fails is that the backing (xfs) filesystem was not formatted with d_type support ("ftype=1"); you can find a discussion on github; https://github.com/docker/docker/issues/27358.
To verify if d_type support is available on your system, check the output of docker info;
Server Version: 1.13.1
Storage Driver: overlay
Backing Filesystem: xfs
Supports d_type: false
Logging Driver: json-file
This requirement is also described in the release notes for RHEL/CentOS
Note that XFS file systems must be created with the -n ftype=1 option enabled for use as an overlay. With the rootfs and any file systems created during system installation, set the --mkfsoptions=-n ftype=1 parameters in the Anaconda kickstart. When creating a new file system after the installation, run the # mkfs -t xfs -n ftype=1 /PATH/TO/DEVICE command. To determine whether an existing file system is eligible for use as an overlay, run the # xfs_info /PATH/TO/DEVICE | grep ftype command to see if the ftype=1 option is enabled.
To resolve the issue, either;
- re-format the device with
ftype=1
- use a different storage driver. Note that the default device mapper configuration (which uses loopback devices) is not recommended for production use, so requires manual configuration.
For backward-compatibility (older versions of docker allowed running overlay on systems without d_type), docker 1.13 will only log a warning in the daemon logs (https://github.com/docker/docker/pull/27433), but will no longer be supported in a future version.
Was able to get past the issue.
The change log for 1.13 says
"IMPORTANT: On Linux distributions where devicemapper was the default
storage driver, the overlay2, or overlay is now used by default
(if the kernel supports it)."
So I tried putting back devicemapper and it is now working as expected.
