CodePipeline deploy problem after upgrading to 1.20.0

[hirro]

I was puzzled by a strange problem.

I had a working copilot pipeline until 20220715, but after updating a README.md on 20220722 the deployment to my dev environment is now failing with the error:

[/Resources/DynamicDesiredCountFunction/Type/Code/S3Bucket] 'null' values are not allowed in templates (Service: AmazonCloudFormation; Status Code: 400; Error Code: ValidationError; Request ID: *******; Proxy: null)

The --stack.yml file in the failing build that used to contain a proper:

  DynamicDesiredCountFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        ZipFile: |
           "...loads of JS skipped here...

Now instead (empty values in S3Bucket as the error message indicates):

  DynamicDesiredCountFunction:
    Type: AWS::Lambda::Function
    Properties:
      Code:
        S3Bucket:
        S3Key:

I noticed that my buildspec.yml downloads and unpacks the latest copilot-linux and that a new 1.20.0 version was released after the last successful build.

Once I changed fixed the copilot-linux to 1.19.0 things started working again.

There seems to be some breaking change to be aware of.

[Lou1415926]

Hi @hirro ! Thank you for reporting this issue that you've encountered. A quick question: does your buildspec package services via copilot svc package --upload-assets or copilot svc package (without the --upload-assets flag)?

[hirro]

Hi @Lou1415926! Thanks for getting back this quickly :-)
The --upload-assets flag is not used.

[Lou1415926]

Thank you @hirro for the information!

Starting from v1.20, codes for custom resources such as DynamicDesiredCountFunction will be uploaded to the S3 bucket first instead of inlined in CloudFormation template. This is done so that the template doesn't get too verbose and large in size quickly.

While in copilot svc deploy the upload is always performed, in svc package it is only performed when --upload-assets flag is present - that is, it doesn't upload anything unless the flag is used.

Therefore, if you want to update your buildspec so that the --upload-assets flag is present, your pipeline should be able to use v1.20 🤞🏼 !

[lennertcc]

I upgraded my buildspec from copilot v1.12.0 to v1.20.0 and added --upload-assets to my copilot svc package and copilot job package commands.

I now run into a permissions problem:

CodeBuild-apponly-988e1b2f-2e27-4426-8d8d-8c57b4fe1617: digest: sha256:feb4f99aaf9d196637a0b99ce65508ddfe990bdb21fc783204c8dc105d553c3e size: 5550
597 | ✘ upload resources required for deployment for app: upload custom resources for "app": upload custom resource "DynamicDesiredCountFunction": upload manual/scripts/custom-resources/dynamicdesiredcountfunction/acd1f00a18ceccc32a780fb208be61f3f62274d775f987fd9feec37493d9173c.zip to bucket stackset-hello-energy-co-pipelinebuiltartifactbuc-9n9zpnwpclxb: AccessDenied: Access Denied
598 | status code: 403, request id: YA29VEGD36XG4VG1, host id: X8NgsxSsIwt7K1WfUXCC6mO3niJRz+238T2NrcfAgXlaB3InVUtVZXtJT08p87JPse/z/rJ/JTE=
599 | Service: contains no tag, adding the one we created with tag: CodeBuild-apponly-988e1b2f-2e27-4426-8d8d-8c57b4fe1617

[efekarakus]

@lennertcc Hi Lennert! Apologies for the inconvenience 🙇 Would you mind running copilot pipeline deploy as well? maybe there was some change in the pipeline template that's missing in v1.12

[lennertcc]

Hi @efekarakus, I ran copilot pipeline update which seems to be the same, not?

I have tried 3 things now:

• updated the existing buildspec from v1.12 to v1.20 and adding --upload-assets, doesn't work
• updated the existing buildspec from v1.12 to v1.19, runs fine and also solves my initial problem (needed at least 1.16 for that, but that's another story)
• created an entirely new pipeline with v1.20 based on the main branch. Also works (see UPDATE below)

With this last try I run into new problems. Packaging looks just fine, but at the end of the buildspec it goes wrong anyway.

d6745c41-2cbe-4c9e-b5a8-fef0ba8581fe-prod: digest: sha256:959c6ee8d8f825d13a398139c356e6151a0ee57472aa32b61f526f3bb8448785 size: 5560
954 | ✘ get application hello-energy-copilot configuration: get application hello-energy-copilot: RequestError: send request failed
955 | caused by: Post "https://ssm.eu-central-1.amazonaws.com/": read tcp 172.17.0.2:59818->10.0.200.128:443: read: connection reset by peer
956 | Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error.

UPDATE:
The mentioned error appears to be transient. I ran the pipeline again without any changes and now the build stage succeeded.

[efekarakus]

Awesome! I'll close the issue now then :) glad to hear it worked out