`setup` command fails with OIDC role while `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` are set in the environment

[blimmer]

Background

Several of my clients are migrating from using static AWS environment variable credentials to using OIDC roles for improved security.

We recently encountered an issue where if AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY are set in the environment (e.g., via project settings or via contexts used in a job), the setup command does not work as expected.

Repro Code

version: 2.1

orbs:
  aws-cli: circleci/aws-cli@4.1.2

workflows:
  workflow:
    jobs:
      - show-oidc-working-without-env-variables-set
      - show-oidc-issue-with-env-variables-set
      - show-oidc-issue-with-env-variables-set-workaround

jobs:
  show-oidc-working-without-env-variables-set:
    docker:
      - image: cimg/base:2024.01-22.04
    steps:
      - checkout
      - aws-cli/setup:
          profile_name: configured-profile
          role_arn: arn:aws:iam::719208690128:role/cci-oidc
          role_session_name: circleci-${CIRCLE_PROJECT_REPONAME}-oidc
          region: us-west-2
      - run:
          name: Test Profile
          command: aws sts get-caller-identity --profile configured-profile
  show-oidc-issue-with-env-variables-set:
    docker:
      - image: cimg/base:2024.01-22.04
        environment:
          # Imagine these are set in a context, or on the project directly
          AWS_ACCESS_KEY_ID: "key"
          AWS_SECRET_ACCESS_KEY: "secret"
    steps:
      - checkout
      - aws-cli/setup:
          profile_name: configured-profile
          role_arn: arn:aws:iam::719208690128:role/cci-oidc
          role_session_name: circleci-${CIRCLE_PROJECT_REPONAME}-oidc
          region: us-west-2
      - run:
          name: Test Profile
          command: aws sts get-caller-identity --profile configured-profile
  show-oidc-issue-with-env-variables-set-workaround:
    docker:
      - image: cimg/base:2024.01-22.04
        environment:
          # Imagine these are set in a context, or on the project directly
          AWS_ACCESS_KEY_ID: "key"
          AWS_SECRET_ACCESS_KEY: "secret"
    steps:
      - checkout
      - aws-cli/setup:
          profile_name: configured-profile
          role_arn: arn:aws:iam::719208690128:role/cci-oidc
          role_session_name: circleci-${CIRCLE_PROJECT_REPONAME}-oidc
          region: us-west-2

          # This is the workaround - set these variables to an env variable that's _not_ set
          aws_access_key_id: UNSET_ENV_VARIABLE
          aws_secret_access_key: UNSET_ENV_VARIABLE
      - run:
          name: Test Profile
          command: aws sts get-caller-identity --profile configured-profile

You can see this code via blimmer/circleci-bug-reports#1 and https://app.circleci.com/pipelines/github/blimmer/circleci-bug-reports/3/workflows/344fc744-de2f-4750-a384-d8d3e0e7d1dd.

No Environment Variables Set

The show-oidc-working-without-env-variables-set works as expected. The orb assumes the OIDC role and the aws sts get-caller-identity --profile configured-profile step reports the expected results

With Environment Variables Set

The show-oidc-issue-with-env-variables-set job shows the issue reported in this bug report. The aws sts get-caller-identity --profile configured-profile step fails because it's using the fake AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY I'm setting.

However, if these were real static credentials (as would be the case in a real example), the command would have returned the IAM user associated with the static credentials.

Workaround

The show-oidc-issue-with-env-variables-set-workaround step shows a workaround for this bug. If you see the aws_access_key_id/aws_secret_access_key parameters to setup to invalid/unset environment variables, the command works as expected (I discuss why below):

The Issue

The problem is here:

aws-cli-orb/src/commands/setup.yml

Lines 117 to 118 in 587c91f

	AWS_CLI_STR_ACCESS_KEY_ID: <<parameters.aws_access_key_id>>
	AWS_CLI_STR_SECRET_ACCESS_KEY: <<parameters.aws_secret_access_key>>

The configure.sh script relies on these global environment variable parameters instead of those defined for the profile:

aws-cli-orb/src/scripts/configure.sh

Lines 9 to 12 in 587c91f

	if [ -z "$AWS_CLI_STR_ACCESS_KEY_ID" ] && [ -z "${AWS_CLI_STR_SECRET_ACCESS_KEY}" ]; then
	temp_file="/tmp/${AWS_CLI_STR_PROFILE_NAME}.keys"
	. "$temp_file"
	fi

This essentially negates the role assumption and uses the global variables instead. By overriding these parameters to something that's unset, we short-circuit the issue:

aws-cli-orb/src/commands/setup.yml

Lines 41 to 55 in 587c91f

	aws_access_key_id:
	description: |
	AWS access key id for IAM role. Set this to the name of
	the environment variable you will use to hold this
	value, i.e. AWS_ACCESS_KEY.
	type: string
	default: AWS_ACCESS_KEY_ID
	aws_secret_access_key:
	description: |
	AWS secret key for IAM role. Set this to the name of
	the environment variable you will use to hold this
	value, i.e. AWS_SECRET_ACCESS_KEY.
	type: string
	default: AWS_SECRET_ACCESS_KEY

This issue will still exist even with the code on master where @brivu changed the environment variable behaviors (#184).

Expected Behavior

I expect that, even if global AWS_* environment variables are set, if I pass a role_arn/role_session_name, that the specified profile will be configured with the OIDC credentials. This allows for multiple profiles (including the inherent/default profile via environment variables) within the same job.

[chrismo]

Thanks for writing this up! Amazing 💯 - using your workaround now.

[marboledacci]

Hi @blimmer, I opened a PR and was addressing this issue there, but when I wrote the tests to validate this behavior, I'm unable to replicate your problem, are you still having it?

[blimmer]

Hey @marboledacci - thanks for looking into this. Did you try recreating the issue in the same way as blimmer/circleci-bug-reports#1? It looks like I tore down the necessary roles and stuff for that example PR, so it'd take me a bit of work to test it again.

If you've got a test bed to recreate that PR, give it a shot. If not, let me know and I can try to get that working again.

Thanks again!

[marboledacci]

Yes I did create a test scenario authenticating with OIDC without passing any env, and one using the same role passing the env. Both times, the validation showed the role assumed, not the user of the credentials. I was using the latest version of aws-cli.

[blimmer]

Yep, this appears to be fixed with the latest version (4.2.3). I tested it with my existing repro case and it passed: https://app.circleci.com/pipelines/github/blimmer/circleci-bug-reports/5/workflows/8e0d33fe-0f71-4798-b61b-6008adfdd437.