Skip to content

Commit efb8166

Browse files
serverless-momserverless-mom
committed
first commit
1 parent ea421cd commit efb8166

File tree

4 files changed

+259
-0
lines changed

4 files changed

+259
-0
lines changed

‎site/assets/guides/images/canary-deploy-01.png

249 KB
Loading

‎site/assets/guides/images/canary-deploy-02.png

762 KB
Loading

‎site/assets/guides/images/canary-deploy-03.png

117 KB
Loading

‎site/content/guides/canary-deploys.md

Lines changed: 259 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,259 @@
1+
---
2+
title: Canary Deploys With Checkly - A Comprehensive Guide
3+
description: >-
4+
You've deployed a new feature to a subset of users, how do you test and monitor this new feature without it being 'lost in the noise'?
5+
author: Nočnica Mellifera
6+
avatar: 'images/avatars/nica-mellifera.png'
7+
tags:
8+
- FAQ
9+
---
10+
11+
Canary deployments are one of the ways we can release updates without violating our Service Level Agreements (SLAs) with our users. By rolling out new code to a small subset of users and looking at the results, we allow a final testing phase with live users before everyone sees our changes. If problems are detected and fixed before updates roll out to all users, we can generally prevent most users from every knowing that we tried to release broken code.
12+
13+
The reason to use canary deployments is very similar to the reason to run synthetic monitoring with Checkly: our production code running in the real world can encounter problems that can't be forseen, no matter how much pre-release testing we do.
14+
15+
This guide will show you how Checkly can enhance your canary deployments, running synthetic monitors against the canary versions of your services. We'll use headers to set feature flags, environment variables to dynamically shift how we're checking our site, and with the [monitoring as code model](https://www.checklyhq.com/learn/monitoring/monitoring-as-code/) we'll dynamically create new monitors as needed to track how our experiments are performing.
16+
17+
One final note, canary deployments are sometimes called blue-green deployments, though some think the two terms imply slight differences in implementation.
18+
19+
## Our Scenario: Checkly and Canary Deployments
20+
21+
Our site, is releasing major new features and we're running a canary deployment to make sure everything's working. Visitors to the site are randomly assigned to a canary group, and receive an additional header on their page requests that activates the new features. Essentially our canary deploy is setting this feature flag to 'true' for a number of users.
22+
23+
We're already monitoring the site with Checkly's browser and API synthetics monitors, and using uptime monitoring to make sure that every URL is available. If we change nothing about our Checkly configuration, some check runs will be randomly assigned to the canary group, and the rest will run with the same set of features as most visitors. We'd like to enhance this experience during our canary deployment. Here are the requirements:
24+
25+
* For better reliability, we want active control whether a check runs as a user that can see the canary deployment.
26+
27+
* Our engineers want to integrate our Checkly monitors into their deploy process, letting us roll back deployments if monitors fail.
28+
29+
* For some checks, we want a version that reports separate data for just this canary deployment, with a chart on our Checkly dashboard showing how checks in the canary group are performing.
30+
31+
At Checkly, we're on a mission to empower developers, making it easier to detect, communicate, and resolve problems before they affect users.
32+
33+
These improvements will help us better **detect** problems with our canary deployment, with clearer ways to **communicate** the results of checks and their context to our team. With clear results information, our team can **resolve** issues faster, making every canary deployment a smoother experience for engineers and users.
34+
35+
## 1. Set Feature Flags With Headers & User Agents
36+
37+
In our scenario, we can control whether we get the canary version of our service with a feature flag. By control a request's headers, we can set the user agent or add arbitrary header values to our requests. Let's set some headers in an API check, a browser check, and a more complex Multistep API check.
38+
39+
### A. Set Headers for an API Check
40+
When running API checks, headers are part of your configuration. You can add [HTTP headers](https://www.checklyhq.com/docs/api-checks/request-settings/#headers) in the Checkly web UI when setting up your API checks, but since we developers prefer to use monitoring as code, here's how to control API checks' headers right from your IDE. API checks are created as a [Checkly construct](https://www.checklyhq.com/docs/cli/constructs-reference/#apicheck), and we can create a new one by creating a new file in our Checkly Project (if you haven't set up a Checkly Project or used monitoring as code before, check out our [getting started tutorial](https://www.checklyhq.com/guides/startup-guide-detect-communiate-resolve/) before returning here to start modifying headers):
41+
42+
```ts {title="api-canary-headers.check.ts"}
43+
import { ApiCheck, AssertionBuilder } from 'checkly/constructs'
44+
import * as path from 'path'
45+
46+
new ApiCheck('hello-api-canary', {
47+
name: 'Hello API, Canary Headers',
48+
activated: true,
49+
maxResponseTime: 10000,
50+
degradedResponseTime: 5000,
51+
request: {
52+
method: 'POST',
53+
url: 'https://httpbin.org/post',
54+
skipSSL: false,
55+
followRedirects: true,
56+
headers: [
57+
{
58+
key: 'canary',
59+
value: 'true'
60+
}
61+
],
62+
assertions: [
63+
AssertionBuilder.statusCode().equals(200),
64+
AssertionBuilder.jsonBody('$.name').notEmpty(),
65+
AssertionBuilder.headers('strict-transport-security', 'max-age=(\\d+)').greaterThan(10000), //checking the headers of the response
66+
]
67+
}
68+
})
69+
```
70+
Since headers are part of the basic configuration of your check, you can populate headers with environment variables, and edit headers for a multiple checks by adding them to a group. We'll use these techniques in parts 2 and 3, below.
71+
72+
73+
### B. Set Headers in a Browser Check
74+
75+
Headers for request are not part of the basic configuration of browser checks. In part because an automated browser will make many checks as part of a single run, and it wouldn't be clear exactly which requests should have an additional header. It's quite easy to add headers to some or all requests from a check written in playwright, with a specialized version of [request interception](https://www.checklyhq.com/learn/playwright/intercept-requests/). In this example we only want to modify requests for SVG files with our new header:
76+
77+
```ts {title="book-listing-canary.spec.ts"}
78+
import { test, expect } from '@playwright/test'
79+
80+
test.describe('Danube WebShop Book Listing', () => {
81+
test('Books Listing', async ({ page }) => {
82+
await page.route('**/*.svg', async route => { //any route ending in .svg
83+
const headers = route.request().headers();
84+
headers['canary'] = 'true'; //add a property
85+
await route.continue({ headers }); //let the request continue with the updated headers
86+
});
87+
await page.goto('https://danube-web.shop/') //the header will be added to any requests with a *.svg route
88+
89+
//Check listed books
90+
91+
})
92+
})
93+
```
94+
95+
By adding a call to page.route we ensure that every single request within this page is modified. Alternatively we could [manage `browser.context()` so that only some requests have these new headers](https://www.checklyhq.com/docs/browser-checks/multiple-tabs/). Note that using browser multiple browser tabs with different `browser.newContext()` calls may have [unexpected effects on your capture of traces and video from your check runs](https://www.checklyhq.com/blog/playwright-video-troubleshooting/), which is why we've gone with the `await page.route()` method here.
96+
97+
The example above filters for all requests with a route ending in `.svg` but you can also filter by resource type, method, post data and more. A full list of the properties available on a request is in [the Playwright documenation](https://playwright.dev/docs/api/class-request).
98+
99+
> Note: in this example we've modified the headers of our outgoing request, and in the context of feature flags and canary deploys there's no reason we'd want to modify the response's headers, but just in case you've come to this page looking for how to modify the response in Playwright, you would add `route.fulfill({header:value})`. The process is documented [on our Learn site under 'response interception.'](https://www.checklyhq.com/learn/playwright/intercept-requests/#response-interception)
100+
101+
102+
### C. Set Headers in a Multistep API Check
103+
A multistep check uses Playwright scripting to perform a series of API requests with more complex evaluation of the response. Since we're only making API requests, we can feed in new headers as a property.
104+
105+
In this example we are making a pair of sequential API checks, the second test step relying on the results of the first step. Both requests use our header settings.
106+
107+
```ts {title="multistep-api-canary.spec.ts}
108+
import { test, expect } from '@playwright/test'
109+
110+
//we use the SpaceX API here as its complex enough to warrant a Multistep API check
111+
const baseUrl = 'https://api.spacexdata.com/v3'
112+
113+
test('space-x dragon capsules', async ({ request }) => {
114+
const headers = {
115+
'canary': 'true'
116+
};
117+
const [first] = await test.step('get all capsules', async () => {
118+
// add our headers to this request
119+
const response = await request.get(`${baseUrl}/dragons`, { headers })
120+
expect(response).toBeOK()
121+
const data = await response.json()
122+
expect(data.length).toBeGreaterThan(0)
123+
return data
124+
})
125+
126+
await test.step('get single dragon capsule', async () => {
127+
const response = await request.get(`${baseUrl}/dragons/${first.id}`, { headers })
128+
expect(response).toBeOK()
129+
130+
const dragonCapsule = await response.json()
131+
expect(dragonCapsule.name).toEqual(first.name)
132+
})
133+
})
134+
```
135+
Note that to create a Multistep API check in your own project you'll also need to create a `.check.ts` file, but this is left out of this guide for brevity, since there don't need to be any settings in the [MultiStepCheck construct](https://www.checklyhq.com/docs/cli/constructs-reference/#multistepcheck) to add canary headers.
136+
137+
These methods let you add canary headers to any Checkly monitor. Next we can make sure that these modified headers look correct for each of our monitors. Let's run our checks once with `npx checkly test`
138+
139+
### D. Check our Modified Headers
140+
141+
In our example scenario our header configuration is quite simple, with just a static key-value pair. But in a real world scenario we might be adding multiple values to a header, possibly with dynamic values as part of an [authentication](https://www.checklyhq.com/learn/playwright/authentication/#handling-authentication-flows) process. As such it's important to check that our updated headers are working before deploying our monitors. We'll run `checkly test` with the `--record` flag to get a report on our check runs:
142+
143+
```bash
144+
npx checkly test --record
145+
```
146+
147+
For our demo we have no other monitors in the `/__checks__` directory, so the three monitors we created will be all that is run (again, if you are feeling lost as we add monitors by creating new files, and manage our project, check out the [beginner's guide to Checkly and monitoring as code](https://www.checklyhq.com/guides/startup-guide-detect-communiate-resolve/)).
148+
149+
```bash
150+
➡️ npx checkly test --record
151+
Parsing your project... ✅
152+
153+
Validating project resources... ✅
154+
155+
Bundling project resources... ✅
156+
157+
Running 3 checks in eu-west-1.
158+
159+
__checks__/api-check-newstyle.check.ts
160+
✔ Hello API, Canary Headers (286ms)
161+
__checks__/book-listing-canary.spec.ts
162+
✔ book-detail-listing.spec.ts (6s)
163+
__checks__/multistep-api-canary.check.ts
164+
✔ Multi-step API Canary Check (586ms)
165+
166+
3 passed, 3 total
167+
168+
Detailed session summary at: https://chkly.link/[yourUniqueLink]
169+
```
170+
171+
At the unique link provided we can see all our check runs succeeded.
172+
173+
![a checkly one-off report](/guides/images/canary-deploy-01.png)
174+
175+
And you can click on each to view the report on the check run. Our browser check will show a full trace, and we can look at our request headers on the network tab:
176+
177+
![a checkly one-off report](/guides/images/canary-deploy-02.png)
178+
179+
For both API monitors and Multistep Monitors, the headers on request and response are listed in the Checkly web UI
180+
181+
![a checkly one-off report](/guides/images/canary-deploy-02.png)
182+
183+
## 2. Integrate Check Runs With CI/CD - Deploy After a Succesful Check Run
184+
185+
Checkly integrates with your CI/CD workflow, and in this example we want to run checks directly after a canary deployment. We are going to feed environment variables to our monitors so that they know to run with a feature flag.
186+
187+
We aren't going to wait for a subset of users to try out the canary deployment, rather we'll just run our Checkly monitors against the Canary deployment, and deploy to production if all checks pass.
188+
189+
190+
```yml {title="checkly-canary-deploy.yml"}
191+
name: 'Checkly Canary Run'
192+
193+
# Set the necessary credentials and export variables we can use to instrument our check run. Use the ENVIRONMENT_URL
194+
# to run your checks against your canary environment
195+
env:
196+
CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
197+
CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
198+
ENVIRONMENT_URL: ${{ github.event.deployment_status.environment_url }}
199+
CHECKLY_TEST_ENVIRONMENT: ${{ github.event.deployment_status.environment }}
200+
CHECKLY_CANARY_RUN: 'TRUE'
201+
jobs:
202+
test-canary:
203+
name: Test E2E for our Canary environment on Checkly
204+
runs-on: ubuntu-latest
205+
timeout-minutes: 10
206+
steps:
207+
- uses: actions/checkout@v3
208+
with:
209+
ref: "${{ github.event.deployment_status.deployment.ref }}"
210+
fetch-depth: 0
211+
- uses: actions/setup-node@v3
212+
with:
213+
node-version-file: '.nvmrc'
214+
- name: Restore or cache node_modules
215+
id: cache-node-modules
216+
uses: actions/cache@v3
217+
with:
218+
path: node_modules
219+
key: node-modules-${{ hashFiles('package-lock.json') }}
220+
- name: Install dependencies
221+
if: steps.cache-node-modules.outputs.cache-hit != 'true'
222+
run: npm ci
223+
- name: Run checks # run the checks passing in the ENVIRONMENT_URL and the IS_CANARY flag, and recording a session.
224+
id: run-checks
225+
run: npx checkly test -e IS_CANARY=${{ env.CHECKLY_CANARY_RUN }} ENVIRONMENT_URL=${{ env.ENVIRONMENT_URL }} --reporter=github --record
226+
- name: Create summary # export the markdown report to the job summary.
227+
id: create-summary
228+
run: cat checkly-github-report.md > $GITHUB_STEP_SUMMARY
229+
- name: Deploy Production # if the check run was successful and we are on Test, deploy to Production
230+
id: deploy-production
231+
if: steps.run-checks.outcome == 'success' && github.event.deployment_status.environment == 'Test'
232+
run: production-deploy # Your production deploy command
233+
```
234+
235+
This check run will start with `IS_CANARY` set to `'TRUE'` and possibly a special `ENVIRONMENT_URL` for our canary deployment. Lets modify `book-detail-canary.spec.ts` from our examples above to work as either a regular production check, or a canary check that sets a special header and checks a different URL when being run as part of a canary deployment.
236+
237+
```ts {title="book-detail.spec.ts"}
238+
import { test, expect } from '@playwright/test'
239+
240+
test.describe('Danube WebShop Book Listing', () => {
241+
test('click book link and verify three elements are visible', async ({ page }) => {
242+
//set variables based on environment values, with defaults
243+
const canary = process.env.IS_CANARY! || 'FALSE'
244+
const baseUrl = process.env.ENVIRONMENT_URL! || 'https://danube-web.shop/'
245+
246+
await page.route('**/*.svg', async route => { //any route ending in .svg
247+
const headers = route.request().headers();
248+
if (canary == 'TRUE'){
249+
headers['canary'] = 'true'; //add a property only if we're testing against a canary deployment
250+
}
251+
await route.continue({ headers });
252+
});
253+
await page.goto(`${baseUrl}category?string=scifi`)
254+
await expect(page.getByText('Laughterhouse-Five')).toBeVisible();
255+
})
256+
})
257+
```
258+
259+
Now we can run this same monitor against both our main line production environments and our canary deployments, with its behavior determined by environment variables.

0 commit comments

Comments
 (0)