Customer issue: STS Assume Role signature mismatch

[skotambkar]

Customer reported signature mismatch error when assuming role via STS, need to investigate the same.

Customer issue: #883 (comment)

Reproducible code as provided by customer: #883 (comment)

[gmauleon]

Just for information, I got the same problem with v0.30.0 on eks calls.

[jasdel]

@gmauleon are you consistently running into this issue or is it periodically occurring? In addition, what AWS region is your application configured for when it encounters this error?

[gmauleon]

Consistent on my side. I shelved it and planned to use v1 for the time being but could certainly retry it this week if you need more informations.

[jasdel]

Thanks for the update @gmauleon. I'm trying to reproduce this issue, but not able to with latest release version v0.30.0, and HEAD of the v2 SDK repo.

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
	"github.com/aws/aws-sdk-go-v2/service/sts"
)

func main() {
	if err := do(); err != nil {
		log.Fatal(err)
	}
}

func do() error {
	cfg, err := config.LoadDefaultConfig(
		config.WithRegion("us-west-2"),
		config.WithClientLogMode(aws.LogSigning),
	)
	if err != nil {
		return err
	}

	stsclient := sts.NewFromConfig(cfg)

	role := fmt.Sprintf("arn:aws:iam::%s:role/%s", accountID, roleName)
	provider := stscreds.NewAssumeRoleProvider(stsclient, role)

	// SDK tip:
	cfg.Credentials = aws.NewCredentialsCache(provider)
	// SDK v0.30.0
	cfg.Credentials = &aws.CredentialsCache{Provider: provider}

	creds, err := cfg.Credentials.Retrieve(context.Background())
	if err != nil {
		return fmt.Errorf("failed to get credentials, %w", err)
	}

	log.Println("Credentials ", creds.AccessKeyID, creds.Source)

	return nil
}

I'm using the following sample code to reproduce this issue. If you have a sample that is able to reproduce this problem that would be very helpful to investigate this issue.

# Update to the tip of the smithy-go currently, `511e8c85` 
go get github.com/awslabs/smithy-go@ac098bf4

# Update to the tip of the SDK currently, `511e8c85`
go get github.com/aws/aws-sdk-go-v2@511e8c85
go get github.com/aws/aws-sdk-go-v2/service/sts@511e8c85
go get github.com/aws/aws-sdk-go-v2/credentials@511e8c85
go get github.com/aws/aws-sdk-go-v2/config@511e8c85

[jasdel]

The only way I've been able to reproduce this is when the original LoadDefaultConfig is unable to retrieve credentials to do the assume role with. Could you look if the cfg.Credentials value is a aws.AnonymousCredentials, or nil?

[gmauleon]

Thanks a lot for looking into this, I'll be out and will not be able to look at it until most likely January, sorry about that :( I'll see if I can look at my code next week but no promises. Have some happy holidays Jason!

…

On 2020-12-18 3:08 p.m., Jason Del Ponte wrote: The only way I've been able to reproduce this is when the original |LoadDefaultConfig| is unable to retrieve credentials to do the assume role with. Could you looking if the |cfg.Credentials| value is a aws.AnonymousCredentials <https://pkg.go.dev/github.com/aws/aws-sdk-go-v2/aws/#AnonymousCredentials>, or |nil|? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#914 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACMH332WJEZOTE57W452TKDSVOZFFANCNFSM4T2Z7BHA>.

[Chandrian]

Using v0.4, I'm getting the same issue (I think) failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: ..., api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
if I use:

ctx := context.TODO()
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion("us-east-1"),
		//config.WithClientLogMode(aws.LogSigning),
	)
	if err != nil {
		log.Fatal(err)
	}

	stsclient := sts.NewFromConfig(cfg)

	provider := stscreds.NewAssumeRoleProvider(stsclient, *assumeRole.masterARN)

	cfg.Credentials = aws.NewCredentialsCache(provider)

And then directly try to use it:

orgClient := organizations.NewFromConfig(cfg)
listAccountsForParentInput := &organizations.ListAccountsForParentInput{
		ParentId: aws.String(ou),
	}
	p := organizations.NewListAccountsForParentPaginator(orgClient, listAccountsForParentInput)
	var emails []string
	for p.HasMorePages() {
		listAccountsForParentOutput, err := p.NextPage(context.Background())
		if err != nil {
			log.Panicf("failed to get the page for accounts in assume OU: %v", err)
		}
	}

But if in between I add the following, everything works fine.... Any idea why?

creds, err := cfg.Credentials.Retrieve(context.Background())
	if err != nil {
		log.Fatal(err)
	}

[jasdel]

Thanks for the update @Chandrian I've been able to reproduce the issue with the following example using your provided input.

func main() {
	if err := countBuckets(context.TODO()); err != nil {
		log.Fatal(err)
	}
}

func countBuckets(ctx context.Context) error {
	cfg, err := config.LoadDefaultConfig(ctx,
		config.WithRegion("us-west-2"),
		config.WithClientLogMode(aws.LogSigning|aws.LogRequest|aws.LogResponseWithBody),
	)
	if err != nil {
		return fmt.Errorf("failed to load config, %w", err)
	}

	const role = "arn:aws:iam::<account>:role/S3ReadOnly"

	stsClient := sts.NewFromConfig(cfg)
	provider := stscreds.NewAssumeRoleProvider(stsClient, role)

	cfg.Credentials = aws.NewCredentialsCache(provider)

	s3Client := s3.NewFromConfig(cfg)
	result, err := s3Client.ListBuckets(ctx, nil)
	if err != nil {
		return fmt.Errorf("failed to list buckets, %w", err)
	}
	log.Println("buckets", len(result.Buckets))

	return nil
}

Log output of request

SDK 2021/01/05 14:30:52 DEBUG Request Signature:
---[ CANONICAL STRING  ]-----------------------------
POST
/

amz-sdk-invocation-id:7e5581df-451b-40bc-b597-87980191039a
amz-sdk-request:attempt=1; max=3
content-length:165
content-type:application/x-www-form-urlencoded
host:sts.us-west-2.amazonaws.com
x-amz-date:20210105T223052Z

amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
---[ STRING TO SIGN ]--------------------------------
AWS4-HMAC-SHA256
20210105T223052Z
20210105/us-west-2/sts/aws4_request
01cb54b891c981d067ae340b7466903c08482ec7e1abb98dac5712fb2fb6b60d
-----------------------------------------------------
SDK 2021/01/05 14:30:52 DEBUG Request
POST / HTTP/1.1
Host: sts.us-west-2.amazonaws.com
User-Agent: aws-sdk-go-v2/0.31.0 GOOS/darwin GOARCH/amd64 GO/go1.15.6 sts
Content-Length: 165
Amz-Sdk-Invocation-Id: 7e5581df-451b-40bc-b597-87980191039a
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=<AccessKeyID>/20210105/us-west-2/sts/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-date, Signature=bcf78b6df8908c8ac63073da076a5bc2848fd7e60aba39bfdc96704fe55e9fb9
Content-Type: application/x-www-form-urlencoded
X-Amz-Date: 20210105T223052Z
Accept-Encoding: gzip

SDK 2021/01/05 14:30:52 DEBUG Response
HTTP/1.1 403 Forbidden
Content-Length: 431
Content-Type: text/xml
Date: Tue, 05 Jan 2021 22:30:51 GMT
X-Amzn-Requestid: 085924bc-3838-4f82-9d0a-e00431418df8

<ErrorResponse xmlns="https://sts.amazonaws.com/doc/2011-06-15/">
  <Error>
    <Type>Sender</Type>
    <Code>SignatureDoesNotMatch</Code>
    <Message>The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.</Message>
  </Error>
  <RequestId>085924bc-3838-4f82-9d0a-e00431418df8</RequestId>
</ErrorResponse>
2021/01/05 14:30:52 failed to list buckets, operation error S3: ListBuckets, failed to sign request: failed to retrieve credentials: operation error STS: AssumeRole, https response error StatusCode: 403, RequestID: 085924bc-3838-4f82-9d0a-e00431418df8, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
exit status 1