CryptographicException KeySet does not exists

I am using Thinktecture Identity server and running it in azure. The issue I am having is that sometimes when I read the Federation metadata XML file or while signing in I get this Keyset does not exists CryptoGraphic exception . Now i know these exceptions can be caused if we do not have proper permissions but the thing is that it just happens sometimes and other times it is working fine. I am not sure how to debug this. Can someone please lead me in some direction.

Also if someone has any idea about what kind of certificate is this system assembly trying to read and how it is configured that will be very helpful as well.

at System.Security.Cryptography.Utils.CreateProvHandle(CspParameters parameters, Boolean randomKeyContainer)
at System.Security.Cryptography.Utils.GetKeyPairHelper(CspAlgorithmType keyType, CspParameters parameters, Boolean randomKeyContainer, Int32 dwKeySize, SafeProvHandle&amp; safeProvHandle, SafeKeyHandle&amp; safeKeyHandle)
at System.Security.Cryptography.RSACryptoServiceProvider.GetKeyPair()
at System.Security.Cryptography.RSACryptoServiceProvider..ctor(Int32 dwKeySize, CspParameters parameters, Boolean useDefaultKeySize)
at System.Security.Cryptography.X509Certificates.X509Certificate2.get_PrivateKey()
at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.get_PrivateKey()
at System.IdentityModel.Tokens.X509AsymmetricSecurityKey.GetSignatureFormatter(String algorithm)
at System.IdentityModel.SignedXml.ComputeSignature(SecurityKey signingKey)
at System.IdentityModel.EnvelopedSignatureWriter.ComputeSignature()
at System.IdentityModel.EnvelopedSignatureWriter.OnEndRootElement()
at System.IdentityModel.EnvelopedSignatureWriter.WriteEndElement()
at System.IdentityModel.Metadata.MetadataSerializer.WriteEntityDescriptor(XmlWriter inputWriter, EntityDescriptor entityDescriptor)
at System.IdentityModel.Metadata.MetadataSerializer.WriteMetadataCore(XmlWriter writer, MetadataBase metadataBase)
at System.IdentityModel.Metadata.MetadataSerializer.WriteMetadata(XmlWriter writer, MetadataBase metadata)
at PeachCourt.Auth.API.Protocols.FederationMetadata.WSFederationMetadataGenerator.Generate()
at Test.Auth.API.Protocols.FederationMetadata.FederationMetadataController.&lt;Generate&gt;b__1()
at Test.Auth.API.Helper.Cache.ReturnFromCache[T](ICacheRepository cacheRepository, String name, Int32 ttl, Func`1 action)
at Test.Auth.API.Protocols.FederationMetadata.FederationMetadataController.Generate()

It is trying to read the certificate that you have configured for token signing. The error message typically indicated that the worker process account has no read access to the private key.