Trying to get Spring/Consul/Vault to work together

Question

I'm trying to do something I think is simple. I want to use Consul for configuration, and Vault for secrets.

What I'm looking for is a simple app like this that allows me to get config and services from Consul, and secrets from Vault.

In my example, Consul and Vault are connected, and I can see Vault in the Consul services and key-value store.

In Postman, I can query Vault directly and see my data at {{vault_url}}/v1/secret/interservice-bus/data-categorizer

Right now I cannot see Consul or Vault values in the test app.

My application.yml is simple

spring:
  profiles:
    active: dev
  main:
    banner-mode: "CONSOLE"
  application:
    name: test-consul

logging:
  level:
    org: INFO

The bootstrap.yml is where I'm doing most of my configuration. Some of it is repeated, because Spring complains if different parts are missing. I'm not sure why.

spring:
    profiles: 
        active: vault, dev
    cloud:
        consul:
            host: 127.0.0.1
            port: 8500
            config:
                enabled: true  
            discovery:
                prefer-ip-address: true
        config:
            discovery:
                enabled: true
                service-id: vault 
            server:
                vault:
                    host: localhost
                    port: 8200
                    scheme: http
                    authentication: TOKEN
                    token: 00000000-0000-0000-0000-000000000000
        vault:            
            host: localhost
            port: 8200
            scheme: http
            connection-timeout: 5000
            read-timeout: 15000
            authentication: TOKEN
            config:
                order: -10
            token: 00000000-0000-0000-0000-000000000000
            discovery:
                enabled: true
                service-id: vault

The token is obscured above, but does match the root token for the vault.

Here's my POM

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>edu.dkist</groupId>
    <artifactId>test-consul</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>test-consul</name>
    <description>Small test to see if consul can work with vault</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.5.8.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
        <spring-cloud.version>Dalston.SR4</spring-cloud.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-consul-config</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-starter-consul-all</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-dependencies</artifactId>
                <version>${spring-cloud.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>org.springframework.cloud</groupId>
                <artifactId>spring-cloud-consul-dependencies</artifactId>
                <version>1.2.1.RELEASE</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

And here's the simple app I'm trying to run

package com.example.demo;

import javax.annotation.PostConstruct;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.EnableAutoConfiguration;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.discovery.EnableDiscoveryClient;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.context.annotation.Configuration;

@Configuration
@EnableAutoConfiguration
@RefreshScope
@SpringBootApplication
@EnableDiscoveryClient
public class DemoApplication {
    @Autowired
    org.springframework.cloud.client.discovery.DiscoveryClient discoveryClient;

    @Value("${test}")
    private String testString;


    @Value("${interservice-bus.data-categorizer.username}") 
    private String user;

    @Value("${interservice-bus.data-categorizer.password}")
    private String password;


    @PostConstruct
    public void setRabbitMQLocation() {
            if (discoveryClient == null) {
            System.out.println("******************************ERROR: Discovery Client is null******************************");
            return;
        }
            System.out.println("****************************** Discovery Client is ACTIVE ******************************");

        org.springframework.cloud.client.ServiceInstance serviceInstance =
                discoveryClient.getInstances("interservice-bus")
                        .stream()
                        .findFirst()
                        .orElseThrow(() -> new RuntimeException("ERROR: " + "interservice-bus" + " not found"));
        System.out.println("****************************** ISB is " + serviceInstance.getHost() + ":" + serviceInstance.getPort() + " ******************************");
    }

    public static void main(String[] args) {
        SpringApplication.run(DemoApplication.class, args);
    }
}

It's not finding "test". I am trying to pull the username and password from Vault, and test from Consul.

I'm getting

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2018-06-20 16:30:33.580 ERROR 92722 --- [           main] o.s.boot.SpringApplication               : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'scopedTarget.demoApplication': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'test' in value "${test}"

What I'm looking for is a simple app like this that allows me to get config and services from Consul, and secrets from Vault.

From searching on the net and looking at examples, it's not clear to me if I need to pull in the Vault dependencies as well or if I'm getting everything from Consul (and thus don't need the Vault dependencies).

Answer 1

To get configuration from Vault you need to add Spring Cloud Vault to your pom.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>

With Spring Cloud Dalston.SR4 (which you used in your pom) you will receive Spring Cloud Vault 1.0.2. So be aware that this version has less stuff than newer one (e.g. you cannot use discovery feature). If you want to use it, consider using at least Edgware.SR4 which comes with Spring Cloud Vault 1.1.1.RELEASE. You can always check version information here: https://github.com/spring-projects/spring-cloud/wiki#release-notes. Anyway, it still will work, but you need to configure it. Your bootstrap.yml should look like this.

spring:
  application:
    name: test-consul
  profiles:
    active: dev
  vault:
    fail-fast: true
    host: localhost
    port: 8200
    scheme: http
    authentication: TOKEN
    config:
      order: -10 # not sure that you need this, didn't investigate
    token: <your_token>

Spring Cloud Vault will search configuration in Vault based on your application name and profile using Vault HTTP API. So you need to store information according to next patterns. (See documentation for Spring Cloud Vault 1.0.2)

/secret/{application}/{profile}
/secret/{application}
/secret/{default-context}/{profile}
/secret/{default-context}

Example:

vault write secret/test-consul/dev username=test_user

In this case during application start Spring Cloud Vault will get properties from secret/test-consul/dev and create Property Source from them. To get value of this test property you should use only key. Example:

@Value("${username}")
private String user;  // will be resolved to "test_user"

Getting properties from Consul KV storage is similar. bootstrap.yml file is:

spring:
  profiles:
    active: dev
  application:
    name: test-consul
  cloud:
    consul:
      host: localhost
      port: 8500
      config:
        enabled: true

Spring Cloud Consul will use next patterns (all of them) for getting key-values from Consul using HTTP API. See documentation for Spring Cloud Consul 1.2.1 (it comes with Dalston.SR4):

config/testApp,dev/
config/testApp/
config/application,dev/
config/application/

So you need to save data accordingly. I did it from Consul GUI by creating folders in KV Storage, using console it should look similarly to this:

consul kv put config/test-consul/test testvalue

By getting data using HTTP API Spring Cloud Consul will create a property source from properties stored there and you will be able to get this using

@Value("${test}")
private String testString; // will be resolved to "testvalue"

You can combine 2 configurations and use Spring Cloud Consul and Vault simultaneously. I did't use application.yml at all for this demo.

Hope this long post will help you :)