Identity Server by leastprivilege doesn't work

I am trying to implement an architecture that follows the OAUTH2/OIDC protocol. In order to do that, I have STS(Identity Server v3 by leastprivilege), ASP.NET WebApi and ASP.NET MVC application for a client. My goal was to have the STS and REST service hosted on Azure so different clients can use them as public services. So far so good. Everything seemed to work smoothly and perfectly before I decided to add a new client that uses one of the redirection flows - Authorization Code flow. I wanted to take advantage of the refresh token option that it offers. I wanted to serve short life access tokens(10 minutes) to that client and make him use a refresh token in order to obtain new tokens. This is how it all looks in code:

STS:

new Client 
{
    ClientId = "tripgalleryauthcode",
    ClientName = "Trip Gallery (Authorization Code)",
    Flow = Flows.AuthorizationCode, 
    AllowAccessToAllScopes = true,
    RequireConsent = false,

    RedirectUris = new List<string>
    {
        Tripgallery.Constants.TripgalleryMvcAuthCodePostLogoutCallback
    },           

    ClientSecrets = new List<Secret>()
    {
        new Secret(Tripgallery.Constants.TripgalleryClientSecret.Sha256())
    },

    // refresh token options
    AccessTokenType = AccessTokenType.Jwt,
    AccessTokenLifetime = 600,
    RefreshTokenUsage = TokenUsage.OneTimeOnly, // Every time generates new refresh token. Not only access token.
    RefreshTokenExpiration = TokenExpiration.Sliding,
    SlidingRefreshTokenLifetime = 1296000,

    PostLogoutRedirectUris = new List<string>()
    {
        Tripgallery.Constants.TripgalleryPostLogoutCallback
    }
}

Mvc application(Client):

private ObjectCache _cache;
private readonly string tokensCacheKey = "Tokens";

public HomeController()
{
    _cache = MemoryCache.Default;
}

// GET: Home
public ActionResult Index()
{
    var authorizeRequest = new AuthorizeRequest(Constants.BoongalooSTSAuthorizationEndpoint);

    var state = HttpContext.Request.Url.OriginalString;

    var url = authorizeRequest.CreateAuthorizeUrl(
    "tripgalleryauthcode",
    "code",
    "openid profile address tripgallerymanagement offline_access",
    Constants.TripgalleryMvcAuthCodePostLogoutCallback,
    state);

    HttpContext.Response.Redirect(url);
    return null;
}

public async Task<ActionResult> StsCallBackForAuthCodeClient()
{
    var authCode = Request.QueryString["code"];

    var client = new TokenClient(
    Constants.TripgallerySTSTokenEndpoint,
    "tripgalleryauthcode",
    Constants.TripgalleryClientSecret
    );

    var tokenResponse = await client.RequestAuthorizationCodeAsync(
    authCode,
    Constants.TripgalleryMvcAuthCodePostLogoutCallback
    );

    this._cache[this.tokensCacheKey] = new TokenModel()
    {
        AccessToken = tokenResponse.AccessToken,
        IdToken = tokenResponse.IdentityToken,
        RefreshToken = tokenResponse.RefreshToken,
        AccessTokenExpiresAt = DateTime.Parse(DateTime.Now.AddSeconds(tokenResponse.ExpiresIn).ToString(CultureInfo.InvariantCulture))
    };

    return View();
}

public ActionResult StartCallingWebApi()
{
    var timer = new Timer(async (e) =>
    {
        var cachedStuff = this._cache.Get(this.tokensCacheKey) as TokenModel;
        await ExecuteWebApiCall(cachedStuff);
    }, null, 0, Convert.ToInt32(TimeSpan.FromMinutes(20).TotalMilliseconds));

    return null;
}

private async Task ExecuteWebApiCall(TokenModel cachedStuff)
{
    // Ensure that access token expires in more than one minute
    if (cachedStuff != null && cachedStuff.AccessTokenExpiresAt > DateTime.Now.AddMinutes(1))
    {
        await MakeValidApiCall(cachedStuff);
    }
    else
    {
        // Use the refresh token to get a new access token, id token and refresh token
        var client = new TokenClient(
            Constants.TripgallerySTSTokenEndpoint,
            "tripgalleryauthcode",
            Constants.TripgalleryClientSecret
        );

        if (cachedStuff != null)
        {
            var newTokens = await client.RequestRefreshTokenAsync(cachedStuff.RefreshToken);

            var value = new TokenModel()
            {
                AccessToken = newTokens.AccessToken,
                IdToken = newTokens.IdentityToken,
                RefreshToken = newTokens.RefreshToken,
                AccessTokenExpiresAt =
                    DateTime.Parse(
                        DateTime.Now.AddSeconds(newTokens.ExpiresIn).ToString(CultureInfo.InvariantCulture))
            };

            this._cache.Set(this.tokensCacheKey, (object)value, new CacheItemPolicy());

            await MakeValidApiCall(value);
        }
    }
}

The problem is that if I have the STS hosted on Azure, for some reason, if I decide to use the refresh token in 20 or more minutes after the access token was expired I get an error. No matter that my refresh token life time is 15 days.

That is the log generated by the STS:

w3wp.exe Warning: 0 : 2017-04-06 12:01:21.456 +00:00 [Warning] AuthorizationCodeStore not configured - falling back to InMemory
w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] TokenHandleStore not configured - falling back to InMemory
w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] ConsentStore not configured - falling back to InMemory
w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] RefreshTokenStore not configured - falling back to InMemory
w3wp.exe Information: 0 : 2017-04-06 12:01:22.371 +00:00 [Information] Start token request
w3wp.exe Information: 0 : 2017-04-06 12:01:22.418 +00:00 [Information] Client secret id found: "tripgalleryauthcode"
w3wp.exe Information: 0 : 2017-04-06 12:01:22.418 +00:00 [Information] Client validation success
w3wp.exe Information: 0 : 2017-04-06 12:01:22.418 +00:00 [Information] Start token request validation
w3wp.exe Information: 0 : 2017-04-06 12:01:22.433 +00:00 [Information] Start validation of refresh token request
w3wp.exe Warning: 0 : 2017-04-06 12:01:22.574 +00:00 [Warning] "Refresh token is invalid"
 "{
  \"ClientId\": \"tripgalleryauthcode\",
  \"ClientName\": \"Trip Gallery (Authorization Code)\",
  \"GrantType\": \"refresh_token\",
  \"RefreshToken\": \"140cfb19405a6a4cbace29646751194a\",
  \"Raw\": {
    \"grant_type\": \"refresh_token\",
    \"refresh_token\": \"140cfb19405a6a4cbace29646751194a\"
  }
}"
w3wp.exe Information: 0 : 2017-04-06 12:01:22.590 +00:00 [Information] End token request
w3wp.exe Information: 0 : 2017-04-06 12:01:22.590 +00:00 [Information] Returning error: invalid_grant
w3wp.exe Information: 0 : 2017-04-06 12:01:29.465 +00:00 [Information] Start discovery request
w3wp.exe Information: 0 : 2017-04-06 12:01:29.512 +00:00 [Information] Start key discovery request

The same case with the STS running on my local machine works as expected. I can get the new tokens with my refresh token.

RESLOVED: The issue really was what Fred Han - MSFT pointed out. I needed to implement persistent store for my refresh tokens. It is really easy to achieve it. This is how I did it:

Startup.cs of the Identity Server :

var idServerServiceFactory = new IdentityServerServiceFactory()
                                .UseInMemoryClients(Clients.Get())
                                .UseInMemoryScopes(Scopes.Get());

//...

// use custom service for tokens maintainance
var customRefreshTokenStore = new CustomRefreshTokenStore();
idServerServiceFactory.RefreshTokenStore = new Registration<IRefreshTokenStore>(resolver => customRefreshTokenStore);

var options = new IdentityServerOptions
{
    Factory = idServerServiceFactory,

    // .....

}

idsrvApp.UseIdentityServer(options);

CustomRefreshTokenStore.cs

public class CustomRefreshTokenStore : IRefreshTokenStore
{
    public Task StoreAsync(string key, RefreshToken value)
    {
        // code that uses persitant storage mechanism
    }

    public Task<RefreshToken> GetAsync(string key)
    {
        // code that uses persitant storage mechanism
    }

    public Task RemoveAsync(string key)
    {
        // code that uses persitant storage mechanism
    }

    public Task<IEnumerable<ITokenMetadata>> GetAllAsync(string subject)
    {
        // code that uses persitant storage mechanism
    }

    public Task RevokeAsync(string subject, string client)
    {
        // code that uses persitant storage mechanism
    }
}

w3wp.exe Warning: 0 : 2017-04-06 12:01:21.456 +00:00 [Warning] AuthorizationCodeStore not configured - falling back to InMemory

w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] TokenHandleStore not configured - falling back to InMemory

w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] ConsentStore not configured - falling back to InMemory

w3wp.exe Warning: 0 : 2017-04-06 12:01:21.512 +00:00 [Warning] RefreshTokenStore not configured - falling back to InMemory

It seem that you store/maintain data in memory, which could be the cause of issue if you host it on Azure website with multi-instances behind the load balancer. You could try to store data in other data store instead of the in-memory storage.

_cache = MemoryCache.Default;

Besides, you store and retrieve tokensCacheKey via memory in your Web API application, which will not work well in Azure multi-instance web-farm environment. Please store the data in external storage, such as Azure storage, database or Redis cache.