Investigate Website A/B Testing

[joepavitt]

Story

No response

Description

A/B Testing in PostHog is known as "Feature Flags" - https://posthog.com/manual/feature-flags

Given we generate static content through Eleventy, we need to investigate what we can get away with and our our site would render the Feature Flags accordingly.

This task will be scoped to exploring the options, and testing out results in PostHog, to understand what the workflow looks like.

[joepavitt]

Currently knee deep in learning about Netlify's Edge Functions as Eleventy Edge (https://www.11ty.dev/docs/plugins/edge/) seems an option, although still trying to understand constraints.

Our biggest challenge is that we're generating static content via Eleventy, but need a dynamic variable for each user depending on the response from the PostHog Feature Flag API call https://posthog.com/manual/feature-flags

[joepavitt]

turns out we want "Experiments" in PostHog (https://posthog.com/manual/experimentation) not Feature Flags

[joepavitt]

Having played with this more, these are my findings:

PostHog Experiments

• Experiments in PostHog are very easy to setup, and configure.
• You can define a single primary target metric (e.g. CTA to Join FF Cloud) and it will auto-optimise for that.
• You can also define secondary metrics, which will dashboard alongside your primary to see additional impact of any changes/variants.

Eleventy Limitations

• Our biggest challenge continues to be our use of Eleventy.
• Eleventy is a static site generator, we need dynamic content to render different on a person-by-person basis.

Eleventy Edge Functions (docs)

• Explicitly advertises that it can be used for AB Testing, although with no examples
• Requires Netlify server, we would no longer be able to host in GitHub pages
• No external functionality/injections, everything must be done locally to the edge function, meaning all of the "magic" of Experiments are lost, we'd just be doing a random selection each time.
• Can set cookies (e.g. AB Testing on Netlify Edge) which means we could in theory package over the PostHog experiments stuff, but it's a lot of manual cranking
• Not so easy to format into AB Testing on a line-by-line basis, this generally works better for complete page redirects, the code to include in the templates is {% edge %} content {% endedge %} where you can then access cookies & query params directly inside the template.

Eleventy Shortcodes:

• I thought I was on to something with Async Paired Shortcodes, it's really easy to build and formats nicely, managed to get a working demo with PostHog experiments that looks like this in a template:

{% abtesting "flag", "control"%}
    <h1 class="text-gray-50 max-w-lg m-auto">
        DevOps for Node-RED
    </h1>
{% endabtesting %}
{% abtesting "flag", "test"%}
    <h1 class="text-gray-50 max-w-lg m-auto">
        Run Node-RED in Production
    </h1>
{% endabtesting %}

and then uses the PostHog Experiments call in the shortcode handler:

eleventyConfig.addPairedShortcode("abtesting", async function (content, flag, value) {
        const fFlag = await posthog.getFeatureFlag('test-flag', 'joepavitt@flowforge.com')
        console.log(fFlag)
        if (fFlag === value) {
            return `${content}`
        } else {
            return ''
        }
})

• The limitation comes in two forms:
  • We need to pass the distinctid/userid into the call each time for PostHog, which is a limitation of their library, but allows us to track exactly which users saw which content, and allows for PH to auto-optimise content.
  • This is all still rendered upon server-start, and is not ran on the edge, therefore the same content still gets rendered to everyone (I'd misinterpreted the docs and thought it still ran at edge, only when building it, did I realise that wasn't the case)

Our Options

From my understanding, we are now left with 3 options:

1. No A/B Testing, stay with Eleventy
2. Have "edge" A/B Testing to randomly show different content to different users, then workout posthog's cookie structure and set the relevant cookie client-side, and call to PH to let it know what we have chosen, so that we can store the relevant data. With this option, we can no longer host on GitHub pages, and would need to switch to Netlify.
3. Switch to something like Astro which is also a static site generator, but have JS built in natively, so can run (where we want to) server-side rendered code. It also comes with much better Headless CMS Integration which would mean we could move the editing/maintenance of the Handbook, Docs & Blog away from the website repo. However, this would be at least a few days of work in order to make that transition as it requires a re-write of the website.