kops version 1.21.5 getting STS Error "failed to verify token" #17555
Description
When attempting to upgrade our kops version to 1.21.5 from 1.19.3; however, we are running into the below error:
Aug 12 17:04:32 ip-172-20-20-107 nodeup[1537]: W0812 17:04:32.155659 1537 executor.go:139] error running task "BootstrapClientTask/BootstrapClient" (9m13s remaining to succeed): bootstrap returned status code 403: failed to verify token: received status code 403 from STS:
We have confirmed that there is no SCP, Policy, or Permission Boundary putting a deny on sts:GetCallerIdentity or any other sts action, so we are stuck on what could be returning the 403 error. The working version we currently have is 1.19.3 and due to this STS error we are not able to upgrade to any of the latest versions.
There is no special routing in place, it goes 0.0.0.0/0 to a NAT Gateway and all the ports for outbound are open in the Security Group.
I tried looking through the code, but couldn't exactly determine where this error might be populating at. I also am able to run aws sts get-caller-identity from the instances themselves and it populates the correct information. I even added sts:GetCallerIdentity to the Inline Policy and confirmed this action is allowed using simulate-principal-policy. I have also checked CloudTrail and don't see any errors for AssumeRole on the instance role.
Note: The Master nodes have not been upgraded to version 1.21.5 they are still on version 1.19.3; however, my next tests are going to be to upgrade the master nodes and see if they work then.
I wanted to open this ticket to see if anyone had any other ideas that could be causing this or if the master not being on the upgraded version would cause this even though the error is a 403 coming from STS.