Skip to content

ENH: S3 access using instance role #2621

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 23, 2018
Merged

ENH: S3 access using instance role #2621

Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d0f29dc
avoid much string manip
anibalsolon Jun 19, 2018
ad93735
Try to connect to S3 using IAM Roles
anibalsolon Jun 19, 2018
aab85cb
import botocore
anibalsolon Jun 19, 2018
502b498
fix string condition, move bounded-wannabe method as function
anibalsolon Jul 23, 2018
5abe1c9
privatize function
anibalsolon Jul 23, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 26 additions & 10 deletions nipype/interfaces/io.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -536,19 +536,39 @@ def _fetch_bucket(self, bucket_name):
session = boto3.session.Session(
aws_access_key_id=aws_access_key_id,
aws_secret_access_key=aws_secret_access_key)
s3_resource = session.resource('s3', use_ssl=True)

# Otherwise, connect anonymously
else:
iflogger.info('Connecting to AWS: %s anonymously...', bucket_name)
iflogger.info('Connecting to S3 bucket: %s with IAM role...',
bucket_name)

# Lean on AWS environment / IAM role authentication and authorization
session = boto3.session.Session()
s3_resource = session.resource('s3', use_ssl=True)

s3_resource = session.resource('s3', use_ssl=True)

# And try fetch the bucket with the name argument
try:
self._get_head_bucket(s3_resource, bucket_name)
except Exception as exc:

# Try to connect anonymously
s3_resource.meta.client.meta.events.register(
'choose-signer.s3.*', botocore.handlers.disable_signing)

iflogger.info('Connecting to AWS: %s anonymously...', bucket_name)
self._get_head_bucket(s3_resource, bucket_name)

# Explicitly declare a secure SSL connection for bucket object
bucket = s3_resource.Bucket(bucket_name)

# Return the bucket
return bucket


def _get_head_bucket(self, s3_resource, bucket_name):
Copy link
Member

@effigies effigies Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should probably be a @staticmethod or a helper function outside the class, as it does not use self at all.


import botocore

# And try fetch the bucket with the name argument
try:
s3_resource.meta.client.head_bucket(Bucket=bucket_name)
Expand All @@ -570,8 +590,6 @@ def _fetch_bucket(self, bucket_name):
% (bucket_name, exc)
raise Exception(err_msg)

# Return the bucket
return bucket

# Send up to S3 method
def _upload_to_s3(self, bucket, src, dst):
Expand All @@ -592,10 +610,8 @@ def _upload_to_s3(self, bucket, src, dst):
s3_prefix = s3_str + bucket.name

# Explicitly lower-case the "s3"
if dst.lower().startswith(s3_str):
dst_sp = dst.split('/')
dst_sp[0] = dst_sp[0].lower()
dst = '/'.join(dst_sp)
if dst[:len(s3_str)].lower().startswith(s3_str):
Copy link
Member

@effigies effigies Jul 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since you're already taking the substring, why not:

if dst[:len(s3_str)].lower() == s3_str:

dst = s3_str + dst[len(s3_str):]

# If src is a directory, collect files (this assumes dst is a dir too)
if os.path.isdir(src):
Expand Down