GitLab CI runner - can't access other reposito

Question

After a recent minor 8.x upgrade, I'm unable to execute GitLab CI tests that also fetch another repository. While everything worked previously, now I get the famous Host key verification failed. error message from ssh. What could be the cause of this?

/etc/gitlab-runner/config.toml:

concurrent = 1

[[runners]]
  name = "python-runner@localhost"
  # ...
  executor = "docker"
  [runners.docker]
    image = "edoburu/python-runner"
    privileged = false
    cap_drop = ["DAC_OVERRIDE"]
    volumes = [
        "/cache",
        "/home/deploy/.ssh:/root/.ssh:ro"
    ]
    # ...

As you can see, the .ssh folder is exposed, to give the container a list of all known hosts (/home/deploy/.ssh/known_hosts). This also gives the container a known SSH key, that I've enabled as deployment key in the repository.

However, the build fails nowadays, which it didn't do before:

Obtaining python-extra from [e-mail]:myproject/python-repo.git@889f8fa0fe485d246d106ccee47aa60b2dd2523e#egg=python-extra (from -r src/requirements.txt (line 63))
  Cloning [e-mail]:myproject/python-extra.git (to 889f8fa0fe485d246d106ccee47aa60b2dd2523e) to /builds/myproject/env/src/python-extra
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Command "git clone -q [e-mail]:myproject/python-extra.git /builds/project/env/src/python-extra" failed with error code 128 in None

The .gitlab-ci.yml file contains:

test:
  image: edoburu/python-runner:base
  stage: test
  script:
  - virtualenv --no-site-packages ../env
  - source ../env/bin/activate
  - pip install --exists-action=w -r src/requirements.txt
  - pip install coverage
  - coverage run --source=src --omit='*/migrations/*' ./src/runtests.py -v2
  - coverage report -m

When I enter the container manually however, everything works fine:

[e-mail] ~ $ docker run -it --volume="/home/deploy/.ssh:/root/.ssh:ro" edoburu/python-runner:base /bin/bash
root@feed357355ad:/# ssh [e-mail]
PTY allocation request failed on channel 0
Welcome to GitLab, Anonymous!
Connection to git.example.org closed.
root@feed357355ad:/# git clone [e-mail]:myproject/python-extra.git  
Cloning into 'python-extra'...
remote: Counting objects: 387, done.
remote: Compressing objects: 100% (176/176), done.
remote: Total 387 (delta 215), reused 374 (delta 208)
Receiving objects: 100% (387/387), 5.97 MiB | 0 bytes/s, done.
Resolving deltas: 100% (215/215), done.
Checking connectivity... done.
root@feed357355ad:/# exit
[e-mail] ~ $

Is there anything that GitLab does differently? Maybe assign IP addresses or something else that causes my builds to fail?

Answer 1

Solved, it turns out that --cap-drop=DAC_OVERRIDE prevented to access the volume. Changing it's owner to root solved it.

Found by adding some debugging code to the .gitlab-ci.yml file:

 script:
  - ping -c 1 git.edoburu.nl
  - ssh-keyscan git.edoburu.nl
  - ls -la ~/.ssh/
  - cat ~/.ssh/known_hosts