Background

Currently the bootstrap phase of the cluster leverages a combination of taskfile tasks and bootstrapping helmfiles/templates to bootstrap the cluster.

Additionally flux is configured to always apply crds before regular cluster apps. These are all housed in the bootstrap/crds tree and are managed by a combination of renovate to rev versions and a github workflow to hydrate the CRDs as actual yaml for flux to apply prior to other applications.

The motivation for handling CRDs this way is to allow for a cleaner repository file structure and eschew unsightly 100+ kustomization files all over the repo.

Proposal

Consider refactoring this approach as it relates to CRDs:

• Replace the CRDs approach with a bootstrap step that will leverage helmfile to do a one-time application of only required CRDs as currently referenced in the main bootstrap/crds/kustomization.yaml file. It could look something like this:

# This helmfile is for extracting and installing Custom Resource Definitions (CRDs) from Helm charts.
# It is not intended to be used with helmfile apply or helmfile sync.

helmDefaults:
  args:
    - --include-crds
    - --no-hooks
  postRenderer: bash
  postRendererArgs:
    - -c
    - yq ea -e 'select(.kind == "CustomResourceDefinition")'

releases:
  - name: cloudflare-dns
    namespace: network
    chart: oci://ghcr.io/home-operations/charts-mirror/external-dns
    version: 1.19.0

  - name: external-secrets
    namespace: external-secrets
    chart: oci://ghcr.io/external-secrets/charts/external-secrets
    version: 0.19.2

  - name: envoy-gateway
    namespace: network
    chart: oci://docker.io/envoyproxy/gateway-helm
    version: 1.5.1

  - name: keda
    namespace: observability
    chart: oci://ghcr.io/home-operations/charts-mirror/keda
    version: 2.17.2

  - name: kube-prometheus-stack
    namespace: observability
    chart: oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack
    version: 77.9.1

... And applied using something like this: helmfile --file crds/helmfile.yaml template --include-crds --no-hooks --quiet | yq ea --exit-status 'select(.kind == "CustomResourceDefinition")' - via taskfile automation as currently configured in the existing taskfile bootstrap steps.

This will require all referenced CRDs to be enabled in the values of the actual helmrelease configurations in the cluster files in the kubernetes/ tree.

The result should be a cleaner more streamlined operation such that there should no longer be a need to run a periodic workflow to hydrate the CRDs vendor tree as well as removal of the flux crds step, given that all bootstrap-required CRDs will be applied to the cluster before flux is installed and attempts to install objects dependent on those CRDs.