How to configure terragrunt when terraform modules are in individual repos? #690

dsantanu · 2023-03-27T19:03:50Z

dsantanu
Mar 27, 2023

Dear all,
I had a single terraform repo with all of the modules in there. but now I have started seperatring 'em inn their individual repos and terragrunt fails to run.

Say, I have two modules, named: tf-service-moda and tf-service-modb as two seperate git repos and all of the terragrunt stuff in a diffrent repo, called: tg-modules. modb is dependent on modb. This is sample file-system:

.
├── services
│   ├── tf-service-moda
│   │   ├── file1.tf
│   │   └── file2.tf
│   └── tf-service-modb
│       ├── file1.tf
│       └── file2.tf
└── tg-modules
    ├── global.yaml
    ├── moda
    │   ├── terragrunt.hcl
    │   └── variables.yaml
    └── modb
        ├── terragrunt.hcl
        └── variables.yaml

Below are the sample config files....

global.yaml:

---
module_branch: master
tf_repo_pfx: "https://git-codecommit.us-west-2.amazonaws.com/v1/repos/tf-service"

moda/terragrunt.hcl:

include {
  path = find_in_parent_folders()
}

locals {
  module_name = "moda"

  values = merge(
    yamldecode(file(find_in_parent_folders("global.yaml"))),
    yamldecode(file("variables.yaml")),
  )
}

inputs = merge(
  local.values,
)

terraform {
  source = "${local.values.tf_repo_pfx}-${local.module_name}?ref=${local.values.module_branch}"
}

modb/terragrunt.hcl:

include {
  path = find_in_parent_folders()
}

locals {
  module_name = "modb"

  values = merge(
    yamldecode(file(find_in_parent_folders("global.yaml"))),
    yamldecode(file("variables.yaml")),
  )
}

dependency "moda" {
  config_path = "../moda"
}

dependencies {
  paths = ["../moda"]
}

terraform {
  source = "${local.values.tf_repo_pfx}-${local.module_name}?ref=${local.values.module_branch}"
}

with this, if I run terragrunt with ~~--terragrunt-source /repos/services//modb~~ -terragrunt-source /repos/services//tf-service-modb all work okay but fit I do just terragrunt plan to let it downloaded directly from git, it fails with this:

Santanu Dasd83436b64af6142517947f0b26bb5772d0ab810bWARN[0000] No double-slash (//) found in source URL /v1/repos/tf-service-moda. Relative paths in downloaded Terraform code may not work.  prefix=[/home/santanu/repos/tg-modules/moda]
 ERRO[0001] 1 error occurred:
 	* bad response code: 401
 
 ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

How do I fix this? What am I missing here? really appreciate any help.

-S

Tracked in ticket #110029

Answered by dsantanu

Apr 1, 2023

okay, finally figured out what was overlooked: The value for tf_repo_pfx MUST be prefixed with git::, for Terraform to process arbitrary Git repositories

tf_repo_pfx: "git::https://github.com/dsantanu/tf-sample"

The above repo works just fine, if tf_repo_pfx(in global.yaml) is updated with git:: prefix.

View full answer

dsantanu · 2023-03-28T17:15:46Z

dsantanu
Mar 28, 2023
Author

Any idea(s)/suggestion(s) from any one? Got stuck in the middle with this.

0 replies

denis256 · 2023-03-28T19:32:17Z

denis256
Mar 28, 2023
Collaborator

Hello,
from the description I have a couple of unclear points:

parameter --terragrunt-source ~/repos/services//modb points to modb but services directory contains tf-service-moda/tf-service-modb
which terragrunt and terraform versions are used

Additionally may be helpful to share execution output with --terragrunt-log-level debug to see from where Bad response code: 401 is returned

2 replies

dsantanu Mar 28, 2023
Author

* parameter `--terragrunt-source ~/repos/services//modb` points to `modb` but `services` directory contains `tf-service-moda/tf-service-modb`

yes, that's true.
This is what I configured:

in global.yaml, I have: tf_repo_pfx: "https://git-codecommit.us-west-2.amazonaws.com/v1/repos/tf-service"
in terragrunt.hcl I have a local variable, like: module_name = "modb"
and then this is what is used to form the actual repo name:

terraform {
  source = "${local.values.tf_repo_pfx}-${local.module_name}?ref=${local.values.module_branch}"
}

all mentioned in the code above.

* which terragrunt and terraform versions are used

$ terragrunt -v
terragrunt version v0.40.2

$ terraform -v
Terraform v1.0.0
on linux_arm64

Additionally may be helpful to share execution output with --terragrunt-log-level debug to see from where Bad response code: 401 is returned

Below is the last few lines of log (from the actual code), just before the 401.........

DEBU[0000] run_cmd output: [santanu] prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Found locals block: evaluating the expressions. prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Evaluated 1 locals (remaining 12): values prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Evaluated 8 locals (remaining 4): aws_env_list, owner_id, aws_acc_map, provider_conf, aws_acc_name, aws_env_accs, aws_accounts, aws_env_cidr prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Evaluated 3 locals (remaining 1): aws_acc_id, aws_iam_role, aws_kms_id prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Evaluated 1 locals (remaining 0): aws_role_arn prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Running command: git log -1 --pretty=format:%an prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
Santanu DasDEBU[0000] run_cmd output: [Santanu Das] prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Running command: git log -1 --pretty=format:%H prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
7c2e0f69b47628229eda76707d4142a4815550d2DEBU[0000] run_cmd output: [7c2e0f69b47628229eda76707d4142a4815550d2] prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Included config /home/santanu/repos/SERVICES/tg_roots/znp/terragrunt.hcl has strategy shallow merge: merging config in (shallow). prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
WARN[0000] No double-slash (//) found in source URL /v1/repos/tf-service-genesis. Relative paths in downloaded Terraform code may not work. prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Detected 2 Hooks prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
DEBU[0000] Downloading Terraform configurations from https://git-codecommit.us-east-1.amazonaws.com/v1/repos/tf-service-genesis?ref=master into /home/santanu/Development/tg_cache/EQbW3gt-7jkGYqIPfdP70a1vLyw/Q0DuRCgSvtaPz10VJOtiKvyyvSc prefix=[/home/santanu/repos/SERVICES/tg_roots/znp/genesis/inc]
ERRO[0001] 1 error occurred:
* bad response code: 401

ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

Let me know if it's not enough.
Looks like it doesn'rt understand the ?ref=master part, hence the 401?

-S

dsantanu Mar 29, 2023
Author

* parameter `--terragrunt-source ~/repos/services//modb` points to `modb` but `services` directory contains `tf-service-moda/tf-service-modb`

Actually now I see, what you mean. That was a typo in my OP. I did men to say: --terragrunt-source ~/repos/services//tf-service-modb but typed wrong, that that was my previous directory structure, when all of the modules were in a single repo. Sorry about that. Updated the OP a well.

dsantanu · 2023-03-30T23:28:06Z

dsantanu
Mar 30, 2023
Author

So, seriously!! No one here got a clue what's going on or what am I doing wrong? If no one experienced that then I'msupposedly doing something wrong and I'm eager to know what and where.

1 reply

denis256 Mar 31, 2023
Collaborator

Hi,
it is complicated to troubleshoot without a test repository, I tried to reproduce a similar setup in
https://github.com/denis256/terragrunt-tests/tree/master/discussion-690 but in my setup plan is executed successfully

dsantanu · 2023-04-01T12:23:32Z

dsantanu
Apr 1, 2023
Author

thanks @denis256 for taking time to create a sample repo for the testing.

For this test, I have created three (1x TG; 2x TF) repos now (based on your example), which is the closest representation of my real-life environments.
It's availabe here: https://github.com/dsantanu/tg-sample-module.git

When run it from the scratch, it actually gives totally different error. Looks like TG cannot figure out what to do when TF modules are in individual repos.

The TF repos are here:

but we should be able to do a successful terragrunt plan/apply without locally cloning any TF modules at all, as it should be downloading those from the upstream in the temporary-space, where it should be running terrafrom form.

Stepes to produce:

git clone https://github.com/dsantanu/tg-sample-module.git
cd tg-sample-module/znp/moda/exc/
terragrunt plan --terragrunt-log-level debug

it ends up with:

$ terragrunt plan
WARN[0000] No double-slash (//) found in source URL /dsantanu/tf-sample-moda. Relative paths in downloaded Terraform code may not work.
ERRO[0000] Did not find any Terraform files (*.tf) in /home/santanu/Development/tg_cache/A4hp0XrkkjIoZltQMU0ZxY0nWZE/3yCD4fTkVaH9m1m5bokh-L8W6rg
ERRO[0000] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

looking in A4hp0XrkkjIoZltQMU0ZxY0nWZE/3yCD4fTkVaH9m1m5bokh-L8W6rg, I see the associated tf-module is downloaded as a file instead:

total 176
-rw-rw-r-- 1 santanu santanu    398 Apr  1 12:19 terragrunt.hcl
-rw-rw-r-- 1 santanu santanu 170229 Apr  1 12:19 tf-sample-moda
-rw-rw-r-- 1 santanu santanu     57 Apr  1 12:19 variables.yaml

and ending up with saying: Did not find any Terraform files (*.tf)

What am I doing wrong here?

0 replies

dsantanu · 2023-04-01T14:19:14Z

dsantanu
Apr 1, 2023
Author

okay, finally figured out what was overlooked: The value for tf_repo_pfx MUST be prefixed with git::, for Terraform to process arbitrary Git repositories

tf_repo_pfx: "git::https://github.com/dsantanu/tf-sample"

The above repo works just fine, if tf_repo_pfx(in global.yaml) is updated with git:: prefix.

0 replies

Gruntwork

How to configure terragrunt when terraform modules are in individual repos? #690

Uh oh!

Uh oh!

dsantanu Mar 27, 2023

global.yaml:

moda/terragrunt.hcl:

modb/terragrunt.hcl:

Replies: 5 comments · 3 replies

Uh oh!

dsantanu Mar 28, 2023 Author

Uh oh!

denis256 Mar 28, 2023 Collaborator

Uh oh!

Uh oh!

dsantanu Mar 28, 2023 Author

Uh oh!

Uh oh!

dsantanu Mar 29, 2023 Author

Uh oh!

dsantanu Mar 30, 2023 Author

Uh oh!

denis256 Mar 31, 2023 Collaborator

Uh oh!

Uh oh!

dsantanu Apr 1, 2023 Author

Stepes to produce:

Uh oh!

dsantanu Apr 1, 2023 Author

dsantanu
Mar 27, 2023

Replies: 5 comments 3 replies

dsantanu
Mar 28, 2023
Author

denis256
Mar 28, 2023
Collaborator

dsantanu Mar 28, 2023
Author

dsantanu Mar 29, 2023
Author

dsantanu
Mar 30, 2023
Author

denis256 Mar 31, 2023
Collaborator

dsantanu
Apr 1, 2023
Author

dsantanu
Apr 1, 2023
Author