Unable to retrieve the decryption key

[billcunnien]

Good afternoon,

Recently, we have been getting an occasional error in the IS (v7.2.4/.NET 9) logs:

Failed to deserialize JSON from grant store.
System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.

This error appears to originate from PersistentGrantSerializer.cs:line 103.

We have 3 servers in our load balanced environment. All three show this error at times (not consistently).

I am not sure how to troubleshoot this error. Can you provide some direction for me?

Thanks!
Bill

[wcabus]

This is a data protection issue where the IDataProtector implementation is unable to find the key that was used when protecting the persisted grant data.

Can you share what your data protection setup looks like? It should be the same for all three servers, so that they share the same data protection keys.

[billcunnien]

I have the following setup where I am configuring services:

builder.Services
    .AddDataProtection()
    .SetApplicationName("MyApplication")
    .PersistKeysToFileSystem(new DirectoryInfo(ConfigBuilder.HostingSettings.DataProtectionShare))
    .ProtectKeysWithCertificate(GetCertificate(ConfigBuilder.HostingSettings.CertificateFileName));

The certificate is the same on all three servers.
The path is a physical, shared path on one of the servers which is accessible to all. My appsettings files handle that appropriately.

This has been in place and working for quite some time. Suddenly our 2nd server is unable to find keys. It does throw a slightly different warning and error when I start it up:

2025-08-25 13:57:25.278 -04:00 [ERR] An exception occurred while processing the key element '<key id="******" version="1" />'.
2025-08-25 13:57:25.278 -04:00 [WRN] Key {******} is ineligible to be the default key because its CreateEncryptor method failed after the maximum number of retries.
System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.

The server has been sidelined until I get this one figured out.

Thanks!
Bill

[wcabus]

If you're able to access the different servers, try and see if they happen to run a different version of the .NET or ASP.NET Core runtime. You can list the installed runtimes by running dotnet --list-runtimes in a terminal window.

I would also suggest temporarily lowering the log level for the Microsoft.AspNetCore.DataProtection namespace, to get additional information while the second server is attempting to load and unprotect the key material. It seems to be loading the XML file contents just fine, but fails when reading/deserializing the key material stored within.

This makes me think that the second server is either failing to load the certificate, or failing to use it for an unknown reason. This issue from .NET 9 RC2 feels similar.

[billcunnien]

All three servers are showing 9.0.8. I will review the log level for that namespace and see if that can provide any additional information.

While I noted the problems on the second server initially, further investigation disclosed these same pile of exceptions occur on startup for each server. After the initial exceptions are listed in the log, each server responds properly and seems to work ok.

One thing I was not expecting were nearly 1000 keys in the shared folder. Most of the time, when I reviewed the folder, it had only a few. A large number of keys seems kind of odd to me.

[billcunnien]

Here is an example of recent entries in the log:

2025-08-27 08:31:22.223 -04:00 [ERR] An exception occurred while processing the key element '<key id="******" version="1" />'.
System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
2025-08-27 08:31:22.224 -04:00 [WRN] Key {******} is ineligible to be the default key because its CreateEncryptor method failed after the maximum number of retries.
System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.AuthenticatedEncryption.CngGcmAuthenticatedEncryptorFactory.CreateEncryptorInstance(IKey key)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.DefaultKeyResolver.CanCreateAuthenticatedEncryptor(IKey key, Int32& retriesRemaining)

The specific key ID (masked) is not a file in the keys folder. So, something is holding on to an invalid key and attempting to load it.

Maybe this will help reveal something to you. I am still a bit in the dark on this one.

[wcabus]

The masked key ID isn't necessarily one of the files, it could also appear as one of the XML elements inside a file as <key id="key-guid-here" version="1">.

And the fact that you have a lot of key files may be a side effect of the server failing to read/decrypt an existing key, causing it to generate a new key to use instead. Did you recently update the certificate used to protect the keys by any chance?

The main issue with data protection and the file storage system, is that it isn't great at synchronizing key material creation: all three instances may decide at roughly the same time that they need to generate a new data protection key, causing three new keys to appear rather than just the one.

[billcunnien]

Something is happening during the app startup that attempts to read/find/load these missing keys. There are dozens of WRN/ERR pairs in the logs, then the server starts operating normally.

What is happening during startup? And, I can I get this fixed so the WRN/ERR are no longer getting recorded in the log?

I will review the certificate to see if that was updated without my knowledge.

[billcunnien]

The certificate has been unchanged since Oct 2024.

[billcunnien]

Just to give you some perspective on the problem - the log adds 17,202 lines when the application starts up. These entries consist of WRN/ERR pairs that are similar to the pair posted earlier. That is a LOT of exceptions! I really need to mitigate this.

[billcunnien]

Even in the "good" logging, I occasionally see an entry like this:

2025-08-27 11:21:53.962 -04:00 [ERR] Failed to deserialize JSON from grant store.
System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.get_Descriptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.Key.CreateEncryptor()
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRing.KeyHolder.GetEncryptorInstance(Boolean& isRevoked)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
   at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
   at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json) in /_/identity-server/src/Storage/Stores/Serialization/PersistentGrantSerializer.cs:line 98
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.GetItemByHashedKeyAsync(String hashedKey)
2025-08-27 11:21:53.963 -04:00 [DBG] refresh_token grant with value: ******-1 not found in store.

I would think that an exception like this should also be mitigated. I am just not sure what is causing this stuff.

[wcabus]

This could be an older persisted grant that was protected using a data protection key which is no longer available. Perhaps it is related to the main issue, if the data protection key used for protecting this grant can't be read again during the startup process.

[wcabus]

Would you be able to locate one of the data protection keys that fails to load, by searching for the key ID in the files? I'm mainly interested in the lifetime details of the keys that can't be loaded - the creation, activation and expiration dates.

Perhaps the keys that are causing issues where all created before the last certificate rotation, making them unreadable/unrecoverable since October 2024. That is, unless this issue only recently started appearing: then the issue could still be caused by another change in the system (.NET runtime update most likely)

[billcunnien]

Moving the key essentially deletes it from use by the Identity Server. Anything protected by that particular key will no longer be accessible. Since it throws an exception when attempting to be loaded. my guess is that the key itself is unrecoverable, therefore unusable by Identity Server. For protected data by the faulting key, I assume that the token refresh would merely fail forcing the user to login again starting the process over with a fresh key. What other process are we talking about never recovering from? I am unclear about that.

[wcabus]

If the key itself is unrecoverable, then indeed, there is no need to keep it as the protected data is lost already. I am only suggesting to keep the key anyway, in case the reason why they fail to load is found and they can be loaded again, to minimize the risk of data loss.

That said, the only reason I can think of, is that the key was created using another certificate than the one which is currently in use, even though the creation date of the key was after the certificate rotation date.

If you still have access to the old certificate(s), you could always try to add them using the UnprotectKeysWithAnyCertificate method, in a final attempt to reload them for unprotecting old data.

[billcunnien]

I do not have any other certificate.

Nearly 1000 keys are in the common folder. I do not understand the proliferation of these keys (new key every time the app pool is recycled, maybe?). Nor do I understand the CryptographicExceptions thrown by these keys that are not found. How does IS even know that a specific key exists (grant store?)?

I have IS setup to store grants in the DB. IS pull a grant and fails to deserialize it:

System.Security.Cryptography.CryptographicException: The key {3ffa7b0b-9392-4a24-9ceb-f4596f79bbc4} was not found in the key ring. For more information go to https://aka.ms/aspnet/dataprotectionwarning
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)
   at Microsoft.AspNetCore.DataProtection.DataProtectionCommonExtensions.Unprotect(IDataProtector protector, String protectedData)
   at Duende.IdentityServer.Stores.Serialization.PersistentGrantSerializer.Deserialize[T](String json) in /_/identity-server/src/Storage/Stores/Serialization/PersistentGrantSerializer.cs:line 98
   at Duende.IdentityServer.Stores.DefaultGrantStore`1.GetItemByHashedKeyAsync(String hashedKey)

If I reset IIS, the IS actually tries to load a huge amount of keys (from the file system??) and throws the exception (tons of times):

System.Security.Cryptography.CryptographicException: Unable to retrieve the decryption key.
   at System.Security.Cryptography.Xml.EncryptedXml.GetDecryptionKey(EncryptedData encryptedData, String symmetricAlgorithmUri)
   at System.Security.Cryptography.Xml.EncryptedXml.DecryptDocument()
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.EncryptedXmlDecryptor.Decrypt(XElement encryptedElement)
   at Microsoft.AspNetCore.DataProtection.XmlEncryption.XmlEncryptionExtensions.DecryptElement(XElement element, IActivator activator)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.DeserializeDescriptorFromKeyElement(XElement keyElement)

No clue why.

So confused right now. The system is faulting and I have no idea why. The system logs exceptions and I do not have a clear path to resolution. I only have one certificate and one place to the shared keys between servers.

Is there a better way?

Bill

[wcabus]

I would suggest the following:

• Manually look up every key which is failing to load during app start. Move each of these keys into a new folder to mark them for removal.
• Check if that fixes the problem, meaning that all remaining keys can be loaded. If not, rinse and repeat.
• Finally, when all faulty keys have been removed from the shared path, I would suggest to disable automatic data protection key generation in every IdentityServer instance and use a separate single instance to check if new key material needs to be generated.

Since this clearly is a data protection specific issue and not an IdentityServer one, we unfortunately cannot invest more time in getting to the bottom of this.
If you need more assistance however, we can connect you to one of our partners to help troubleshooting this on a consultancy basis. Feel free to contact us for more details.

[billcunnien]

ok, thanks

[billcunnien]

Thanks for the help!