A/B Testing Infrastructure & Automated Netlify Workflow

[joepavitt]

Description

Continued work from #562 <- contains a lot of detail about the implementation and process that follows.

I needed to pull content into this repo (away from my personal) so that I could run the build actions/workflows for pushing to Netlify.

Didn't make sense to merge this work back into my personal repo, to then continue with #562 so I'm just making this fresh PR.

We already have a deployed version in Netlify running from the live branch: https://flowforge-website.netlify.app/

Netlify Env Var currently points to PostHog's Test Project - would need to switch.

Recommended next steps before review:

• Remove the use of {% abtesting %} from this branch for now, as only contains test experiments.
• Update the Environment Variable in Netlify to point to PostHog "Production" project
• Trigger a manual build workflow to get the latest of this branch running on Netlify

Recommended next steps after review:

• Switch the DNS Settings of https://flowforge-website.netlify.app/ to point to flowforge.com
• Merge this PR into main
• Verify that the merge to main in turn triggers the new build workflow and deploys any new changes to the Netlify Site automatically

Related Issue(s)

Closes #475

Checklist

• I have read the contribution guidelines
• I have considered the performance impact of these changes
• [-] Suitable unit/system level tests have been added and they pass
• [-] Documentation has been updated

[github-actions]

Images automagically compressed by Calibre's image-actions ✨

Compression reduced images by 46.2%, saving 458.49 KB.

Filename	Before	After	Improvement	Visual comparison
src/handbook/images/screenshots/ph-experiment-funnel.png	225.29 KB	108.73 KB	-51.7%	View diff
src/handbook/images/screenshots/ph-experiment-trend.png	240.61 KB	114.42 KB	-52.4%	View diff
src/handbook/images/screenshots/ph-experiment-users.png	525.46 KB	309.71 KB	-41.1%	View diff

176 images did not require optimisation.

[joepavitt]

Actually, thinking about it @knolleary - we could merge this before doing the DNS switch, as all it would do is have the slightly more up to date version running in Netlify, merging this would not affect the existing GH Pages build given we've changed that build.yml workflow

[joepavitt]

When testing locally, @knolleary saw an error upon first time running linked to the edge functions not being available. This was similar to an error I see sometimes when deploying on Netlify within about 20 seconds of the deploy occuring).

We were then unable to reproduced, and with fresh clones and installs on my own machine, I could not reproduce.

I asked @iskerrett to give it a go, and he too hit the same problems as Nick (which were circumvented the second time running). Whilst, for local development it's not too bad, as it just occurs the first ever time someone clones and instals with Edge Functions, it does seem to correlate to the issue I've seen on Netlify too, so this needs further investigation and fixing.

The issues went as follows:

1. When first running, Netlify wants to setup an Edge Functions Environment

2. After this, the server loads up, but the website crashes

3. The reported error says that netlify could not find the _generated netlify functions

My guess here is that because the setup occurs, netlify then doesn't return back to the list of edge functions that we have and run it's auto-generation process (where it creates output into _generated. Upon restarting the server, the environment has been configured for Edge Functions, and so works.

[joepavitt]

just sanity checked this again with a fresh Netlify deploy, no edge function errors, but it is slow to load the first request after a deploy, but that seems linked to how Netlify cache rather than anything specific to Edge Functions

[knolleary]

Some typos but otherwise happy with this.