Bump AWS SDK version to version "~>3".

[AustinTag]

We have a plugin:

https://github.com/awslabs/logstash-output-amazon_es

which we want to bump the aws-sdk version to "~>3". To avoid any dependency conflicts with other logstash plugins, we would also like to bump the version of aws-sdk used by logstash-mixin-aws. My thinking is that this should be a major version bump, and that each of the logstash plugins that are using "logstash-mixin-aws" as a runtime dependency should upgrade their dependency to "logstash-mixin-aws" with a version bump as well.

I am in the process of developing and testing this change, but wanted to create the issue to track any discussion about this.

[AustinTag]

Actually, after making this change locally and testing all of the logstash plugins that use logstash-mixin-aws as a dependency, it appears that upgrading to aws-sdk version "~> 3" is not a breaking change for any of these plugins (that is, all of their tests passed with the upgraded sdk). Therefore it seems safe to simply do a minor version bump in logstash-mixin-aws. Thoughts?

[colinsurprenant]

As mentioned in #39 bumping this dependency version has the side effect of adding ~200 more dependencies and we will defer using v3 of the aws-sdk to our upcoming aws integration plugin (see https://www.elastic.co/blog/logstash-lines-introduce-integration-plugins) where we will be in a better place to only pick the required dependencies.

[HitkoDev]

@colinsurprenant v3 also brings support for web identity provider, which is a crucial security feature when running on AWS EKS.

[AustinTag]

Is there an ETA on the AWS integrations plugin?

[colinsurprenant]

We are actively moving forward with integration plugins but we don't have an ETA.

[JPLachance]

Hello @colinsurprenant,

A colleague did the aws-sdk upgrade and added the support for AssumeRoleWebIdentityCredentials in a fork: master...coveooss:CLOUDINFRA-2050---logstash-in-EKS

Do you want us to submit a pull request?

Thanks!

[colinsurprenant]

@JPLachance Sure! Go ahead! Thanks.

[itssimon]

It looks like bumping the AWS SDK version to at least 2.11.345 would introduce support for IAM roles for service accounts (IRSA). That should be an even smaller change to make, but add heaps of value for AWS EKS users.

Would you consider this as an interim solution?

[dgonzalezruiz]

This has been opened for over a year now 😅 is there any chance we can get upgraded aws-sdk gem soonish? As @itssimon pointed out, latest v2 should already support temporary creds. issued via AssumeRoleWebIdentityCredentials method, which would unblock using logstash with any AWS related plugin, that wants to benefit from reduced complexity handling security credentials (thanks to IRSA).

Logstash integration plugins seem great, but it's been over a year and there's only three of them available (seem like that the same ones made available in the original announcement). What's the best course of action that one could take here @colinsurprenant ?

[dgarbus]

This has been opened for over a year now 😅 is there any chance we can get upgraded aws-sdk gem soonish? As @itssimon pointed out, latest v2 should already support temporary creds. issued via AssumeRoleWebIdentityCredentials method, which would unblock using logstash with any AWS related plugin, that wants to benefit from reduced complexity handling security credentials (thanks to IRSA).

Logstash integration plugins seem great, but it's been over a year and there's only three of them available (seem like that the same ones made available in the original announcement). What's the best course of action that one could take here @colinsurprenant ?

I tried to get this working with v2 of the ruby SDK and had no success. I don't even see a reference to AssumeRoleWithWebIdentity in the v2 source code.

[overskylab]

any updates?

[aackerman]

v2 of the SDK doesn't doesn't support web identity credentials through the CredentialProviderChain so v2 isn't really an option since that's what most open source projects use. The 6.8.13 logstash docker image with the logstash-output-amazon_es plugin uses v2.11.540 and doesn't work with IRSA even though aws-sdk-core is above the documented minimum SDK version.

[karthik-devops-tech]

Any updates on this issue to support IRSA?

[christianherweg0807]

I´ve tested with aws sdk v3, by patching mixin locally...
Everything works fine. @colinsurprenant : the wide range of new dependency comes with the "new" modularity of aws-sdk.
There is a versioned gem for ervery service now.

The pessimistic version locking in this plugin block´s ervery plugin developer from using the new SDK. So please update or change the dependency to >= 2.