Problem:

when using StsAssumeRoleCredentialsProvider, in case of STS or other failure, the client lib applies fallback to the rest of the credential providers in the providers chain

Details

Usage:

sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required \
    awsRoleArn="arn:aws:iam::1234567890:role/msk-client-role" awsRoleSessionName="session"  awsStsRegion="us-west-2" ;

Setup:

• i use k8s
• my deployment uses annotations and kube2iam via IMDS
• annotated role (aka pod's role) doesn't have permissions to access MSK
• my-client-role IAM role is having all permissions for Kafka and must be assumed by pod's role

Failure Scenario:

• during massive scale (huge number of clients hosted on a pod), either STS or kube2iam or IMDS serivce get throttled or a bad performance resulting in Assume Role credentials provider throwing exception
• automatic fallback engages IMDS (because it is part of the default chain)
• pod's role doesn't have permissions for Kafka
• clients getting Access Denied