gitlab-ci cache on kubernetes with minio-service n

Question

I'm running gitlab 10.4.3 with gitlab-runner 10.4.0 as a kubernetes deployment with kubernetes runner and a minio-server for caching. I installed it according to the gitlab docs.

Everything worked with the cache as expected, when adding the appropriate settings to the .gitlab-ci.yml-file:

build:
  stage: build
  variables:
    GIT_STRATEGY: fetch
  cache:
    key: "$CI_BUILD_REF_NAME"
    paths:
      - node_modules/
  script:
    - compile-templates.sh
    - yarn install
    - yarn run build

The pipeline output did fill the cache the first time and subsequent runs on the same branch correctly pulled and pushed the cache:

Running on gitlab/runner-abcdefg-project-123-concurrent-123456 via gitlab-runner-123456-987654...
Cloning repository for feature/gitlab-cache with git depth set to 1...
Cloning into '/group/project'...
Checking out b1348594 as feature/gitlab-cache...
Skipping Git submodules setup
Checking cache for feature/gitlab-cache...
Downloading cache.zip from http://minio-service:9000/runner/runner/abcdefg/project/123/feature/gitlab-cache 
Successfully extracted cache
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
Login Succeeded
[...snip...]
Creating cache feature/gitlab-cache...
node_modules/: found 26281 matching files
Uploading cache.zip to http://minio-service:9000/runner/runner/abcdefg/project/123/feature/gitlab-cache 
Created cache

However, after some runs it suddenly stopped working - and I do not know why. I've also tried a job-global cache-definition to no avail - it seems that the gitlab-runner is simply ignoring the setting and just skips the "Checking cache part":

Running on gitlab/runner-abcdefg-project-123-concurrent-123456 via gitlab-runner-123456-987654...
Cloning repository for feature/gitlab-cache with git depth set to 1...
Cloning into '/group/project'...
Checking out b1348594 as feature/gitlab-cache...
Skipping Git submodules setup
$ docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
Login Succeeded
[...snip...]
Job succeeded

My gitlab-runner config.toml has the appropriate settings:

[[runners]]
  // ...
  [runners.kubernetes]
  // ...
  [runners.cache]
    Type = "s3"
    // I've also tried http://minio-service:9000 to no avail
    ServerAddress = "minio-service:9000"
    AccessKey = "xxxxxxxxxxxxxxxxxxxxxxxx"
    SecretKey = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
    BucketName = "runner"
    Insecure = true

Note: If I change the settings to invalid ones, I do not get any error messages in the runners logs.

The port is reachable from the runner itself and from job pods:

$ curl -s "http://minio-service:9000"
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied.</Message><Key></Key><BucketName></BucketName><Resource>/</Resource><RequestId>12345</RequestId><HostId>12345</HostId></Error>

The bucket on minio exists as checked in the local volume and via the minio client mc.

There are no errors in the logs of the pods of gitlab-runner or the minio-server.

It's just like the cache-settings of the .gitlab-ci.yml are simply ignored and I'm out of ideas on what's left to check. Anyone with any ideas?

Answer 1

So, the problem was an invalid documentation in combination with a silent ignore of invalid cache:keys. If a branch is e.g. named feature/some-new-thing, the resulting key of "$CI_BUILD_REF_NAME" would lead to an invalid cache:key containing a "/"-character - which in turn would render the whole cache-section invalid, but which is silently ignored.

Instead, just use "${CI_BUILD_REF_SLUG}", which will contain an url-safe - and therefore cache:key-safe - version of the branch name:

cache:
  key: "${CI_BUILD_REF_SLUG}"
  paths:
    - node_modules/

A bug report has been opened.