Keycloak - Infinispan Redis cache store

Question

Currently setting up a keycloak cluster in standalone-ha mode, to be able to run on docker swarm. In keycloak, the user sessions are cached in an embedded infinispan store and infinispan can be configured to be a distributed cache across the cluster.

I have also set the owner to be 2, but the problem is that.. during scale-down, there is a possibility for the user-sessions to be lost, If both the owners containing the cache are killed during scale-down.

I have also read about Infinispan Redis cache store, but I am not sure how to configure this.

Question 1: Is it possible to configure Keycloak Infinispan to user a Redis Store ?

Question 2: If this is not possible, is there a way that one could overcome this problem ?

Any suggestions would be helpful.

Answer 1

Due to this PR https://github.com/keycloak/keycloak/commit/056ba75a72b1595ca9fa471f5693201fd5b2c7ae by default (Keycloak latest release 6.0.1) the Infinispan Connection SPI which uses InfinispanChangelogBasedTransaction.java has a very particular use of a CacheDecorator.java which will skipCacheStore. This means that no matter if you configure a store with persistence the store will be ignored.

In order to achieve what you want, besides configuring the store, you would have to customize most of the SPIs here https://github.com/keycloak/keycloak/tree/master/model/infinispan/src/main/resources/META-INF/services in order to make sure that Keycloak will be using the cache store.

This will also not be easy since there are a lot of perks involved into the process, for example, since Keycloak is using the Marshaller of Jboss, if you customize this SPIs you would have to bring most of the org.keycloak.models.sessions.infinispan package and register your module to make sure that Wildfly will be able to see the entities to marshall.

Another thing is that you should, with Redis, configure most of the caches pointing to one common database, except the authenticationSessions which cannot be in the same database as sessions, otherwise, there will be conflicts like the RootAuthenticationSesssionEntity being found but expected to be SessionEntityWrapper.

To resume, the process will be painfull, but if you want to dare and do it, this is how I achieved it:

• Introduced a custom InfinispanConnectionProviderFactory in order to have full capability to use infinispan configuration and then configure my containers like:

private Configuration getRedisConfiguration(int database) {
    ConfigurationBuilder cb = new ConfigurationBuilder();
    cb.persistence()
      .passivation(false)
      .addStore(RedisCacheStoreConfigurationBuilder.class)
      .ignoreModifications(false)
      .fetchPersistentState(false)
      .purgeOnStartup(false)
      .preload(false)
      .shared(true)
      .addProperty("host", System.getenv("REDIS_HOST"))
      .addProperty("port", System.getenv("REDIS_PORT"))
      .addProperty("database", String.valueOf(database));

    return cb.build();
  }

The RedisCacheStoreConfigurationBuilder that you see there is basically a stripped-down version of the original store but I don't need Sentinel or Server mode I just want to connect to a host, port, and database.

Then I basically copied the org.keycloak.models.sessions.infinispan removing everything related to remove cache, and instead of using normally the cache without the decorator to skipCacheStore.

Let me know if I can help with something, I will most prepare a post that instructs more detailed how to do this, involving also a repository that will contain the codes that I am talking about. Please let me know more if someone is still trying this.

Answer 2

Any particular reason for using a Redis store behind Infinispan?

A simpler solution might be to configure persistence to file or shared DB. For a cache use case like this, file based persistence might be enough. See here for example on configuring Infinispan with file based persistence. Alternatively, you can store to shared DB, e.g. Postgresql, but that requires more set up (see ref card for example).

Answer 3

When expecting to scale large systems dynamically, it is expected to avoid such a constraint to register a list of available nodes in configurations. So Redis nodes discovery is a benefit here.

Infinispan itself supports Redis Store as an extension implementing both SPI and adding XML entities to ease configuration:

• http://infinispan.org/docs/cachestores/redis/

• https://github.com/infinispan/infinispan-cachestore-redis

• http://infinispan.org/docs/stable/user_guide/user_guide.html#custom_cache_stores

But this extension is not supported (yet) in WildFly infinispan subsystem - as Keycloak relies on WildFly.

So I expect the following tasks to get Infinispan Redis Store available for WildFly, and so Keycloak:

• Create jboss module for infinispam redis store jar - see modules/system/layers/base/org/infinispan/

• Create a "Custom Cache Store" factory able to instanciate Redis Store objects (store, servers and connection pool) from WildFly configuration key/value properties. This has to be added to WildFly as a jboss module too

• Use WildFly infinispan subsystem "local-cache" "custom" to configure this factory with attributes class and properties: https://wildscribe.github.io/WildFly/11.0/subsystem/infinispan/cache-container/local-cache/store/custom/index.html

Work is in progress and it is possible code and configuration will be published.