terraform-aws-wrangler

Terraform module to collect files from various sources and manage them in an S3 bucket. In order to support arbitrary file-types, this module uses terraform-external-file-cache to create a local cache of the files. The files are then managed in the S3 bucket using the terraform resource aws_s3_object. A sha512 hash of every file is also published to the bucket.

Use Cases

This module supports a couple use cases:

1. Retrieve files from source URIs and store them in an S3 bucket.
2. Copy files from one S3 bucket to another.

Retrieve files and store them in an S3 bucket

These variables are used to retrieve files and store them in an S3 bucket:

• uri_map - Map of URI = S3 Path. Each URI will be retrieved and the file will be saved to S3 Path. The filename is preserved. If the map is empty (the default), no files are retrieved. Here is an example of the structure:

  uri_map = {
    # salt for windows
    "https://packages.broadcom.com/artifactory/saltproject-generic/windows/3006.10/Salt-Minion-3006.10-Py3-AMD64-Setup.exe" = "saltstack/salt/windows/"
  
    # python for windows
    "https://www.python.org/ftp/python/3.6.8/python-3.6.8-amd64.exe" = "python/python/"
  }

• bucket_name - Name of the S3 bucket where files will be stored.

• prefix - S3 prefix prepended to all S3 key paths when the files are put in the bucket.

Copy files from one bucket to another

This is accomplished by getting a list of the s3 objects in the source bucket, and constructing the uri_map. This list can be provided using the data source aws_s3_objects, but when doing so it is recommended to generate that list in a separate state and output the value. This is because the output of a data source or resource cannot be used in the for_each statement of a resource (without encountering chicken/egg problems).

See the s3_sync test for an example.

Testing

Manual testing:

# Replace "xxx" with an actual AWS profile, then execute the integration tests.
export AWS_PROFILE=xxx 
make terraform/pytest PYTEST_ARGS="-v --nomock"

For automated testing, PYTEST_ARGS is optional and no profile is needed:

make mockstack/up
make terraform/pytest PYTEST_ARGS="-v"
make mockstack/clean

Requirements

Name	Version
terraform	>= 1.0
aws	>= 4.0.0

Providers

Name	Version
aws	>= 4.0.0

Resources

Name	Type

Inputs

Name	Description	Type	Default	Required
bucket_name	Name of the S3 bucket where file artifacts are to be stored	string	n/a	yes
create_hashes	Create and host sha512 hashes of each file in the uri_map	bool	true	no
prefix	S3 key prefix to prepend to each object	string	""	no
python_cmd	Command to use with the filecache module when executing python external resources	list(any)	[ "python" ]	no
s3_endpoint_url	S3 API endpoint for non-AWS hosts; format: https://hostname:port	string	null	no
uri_map	Map of URIs to retrieve and the S3 key path at which to store the file	map(string)	{}	no

Outputs

Name	Description
files	Map of file keys => source_hash
hashes	Map of hash keys => source_hash