Skip to content

🌱 Update the dev-test ClusterClass #1848

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 31, 2024
Merged

🌱 Update the dev-test ClusterClass #1848

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
119 changes: 88 additions & 31 deletions docs/book/src/development/development.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,14 @@
- [Using your own capi-openstack controller image for testing cluster creation or deletion](#using-your-own-capi-openstack-controller-image-for-testing-cluster-creation-or-deletion)
- [Building and upload your own capi-openstack controller image](#building-and-upload-your-own-capi-openstack-controller-image)
- [Using your own capi-openstack controller image](#using-your-own-capi-openstack-controller-image)
- [Developing with Tilt](#developing-with-tilt)
- [Testing Cluster Creation using the 'dev-test' ClusterClass with Tilt](#testing-cluster-creation-using-the-dev-test-clusterclass-with-tilt)
- [Developing with Tilt](#developing-with-tilt)
- [Apply ClusterClass and create Cluster](#apply-clusterclass-and-create-cluster)
- [Automatically applying kustomizations with Tilt](#automatically-applying-kustomizations-with-tilt)
- [Using the 'dev-test' ClusterClass without Tilt](#using-the-dev-test-clusterclass-without-tilt)
- [Creating a Kind Cluster](#creating-a-kind-cluster)
- [Secret Configuration](#secret-configuration)
- [Apply the ClusterClass and create Clusters](#apply-the-clusterclass-and-create-clusters)
- [Running E2E tests locally](#running-e2e-tests-locally)
- [Support for clouds using SSL](#support-for-clouds-using-ssl)
- [Support for clouds with multiple external networks](#support-for-clouds-with-multiple-external-networks)
Expand All @@ -15,9 +22,13 @@
- [Create E2E test environment](#create-e2e-test-environment)
- [OpenStack](#openstack)
- [DevStack](#devstack)
- [Server side](#server-side)
- [CAPO side](#capo-side)
- [Running E2E tests using rootless podman](#running-e2e-tests-using-rootless-podman)
- [Host configuration](#host-configuration)
- [Running podman system service to emulate docker daemon](#running-podman-system-service-to-emulate-docker-daemon)
- [API concepts](#api-concepts)
- [`referencedResources`](#referencedresources)

<!-- END doctoc generated TOC please keep comment here to allow auto update -->

Expand Down Expand Up @@ -52,42 +63,28 @@ After generating `infrastructure-components.yaml`, replace the `us.gcr.io/k8s-ar

## Testing Cluster Creation using the 'dev-test' ClusterClass with Tilt

This guide demonstrates how to create a Kubernetes cluster using ClusterClass, specifically designed for a development environment. It includes initializing a Kind cluster, configuring secrets, and applying ClusterClass to deploy your cluster with Tilt.
This guide demonstrates how to create a Kubernetes cluster using a ClusterClass, specifically designed for a development environment. It includes configuring secrets, applying the ClusterClass, and creating a cluster with Tilt.

### Creating a Kind Cluster

Create a Kind cluster. This process involves the initialization of Cluster API providers for OpenStack.
The `dev-test` ClusterClass is designed for development.
This means that it is using the latest (potentially unstable) API version.
The defaults are also aligned with the devstack setup (documented below) to make it as easy as possible to use in a development flow.
However, this also means that it may *not* be well suited for general usage.

```bash
kind create cluster
export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure openstack
```

### Secret Configuration

CAPO needs a clouds.yaml file in order to manage the OpenStack resources needed for the Cluster. This should be supplied as a secret named `${CLUSTER_NAME}-cloud-config`. You can create this secret for example with:

```bash
kubectl create secret generic ${CLUSTER_NAME}-cloud-config --from-file=clouds.yaml
```

You can also make Tilt create it for you when you run `tilt up`, more on that in the next section.

## Developing with Tilt
### Developing with Tilt

We have support for using [Tilt](https://tilt.dev/) for rapid iterative development. Please visit the [Cluster API documentation on Tilt](https://cluster-api.sigs.k8s.io/developer/tilt.html) for information on how to set up your development environment.

Default values align with the devstack setup, as documented below. When specifying the `KUBERNETES_VERSION`, ensure an image named `ubuntu-2204-kube-{{ KUBERNETES_VERSION }}` is available in your environment, corresponding to that Kubernetes version.
For using Tilt with ClusterClass, update your `tilt-settings.yaml` file as described:
The `Tiltfile` in the Cluster API repository can be used as is with CAPO, but we need to add some configuration.
For using Tilt with ClusterClass, update your `tilt-settings.yaml` file (located in the root of the CAPI repository) as described:

```yaml
template_dirs:
openstack:
# Make Tilt aware of the CAPO templates
- ../cluster-api-provider-openstack/templates

kustomize_substitutions:
CLUSTER_TOPOLOGY: "true"
# Define the name of your cluster (e.g., "dev-test")
CLUSTER_NAME: "<cluster_name>"
# Desired Kubernetes Version for Your Cluster (e.g., "v1.28.5")
KUBERNETES_VERSION: "<kubernetes_version>"
# [Optional] SSH Keypair Name for Instances in OpenStack (Default: "")
OPENSTACK_SSH_KEY_NAME: "<openstack_keypair_name>"
# [Optional] Control Plane Machine Flavor (Default: m1.medium)
Expand All @@ -97,10 +94,30 @@ kustomize_substitutions:
# [Optional] OpenStack Cloud Environment (Default: capo-e2e)
OPENSTACK_CLOUD: "<openstack_cloud>"

# [Optional] Automatically apply a kustomization, e.g. for adding the clouds.yaml secret
additional_kustomizations:
secret_kustomization: /path/to/kustomize/secret/configuration
```

### Apply ClusterClass and create Cluster

When you are happy with the configuration, start the environment as explained in the CAPI documentation.
Open the Tilt dashboard in your browser.
After a while, you should be able to find resources called `CAPO.clusterclasses` and `CAPO.templates`.
These shoud correspond to what exists in the `templates` folder and you should see widgets for applying and deleting them.

**Note:** When you apply a cluster template, there will be a `KUBERNETES_VERSION` variable.
This variable is used to pick the image used!
Ensure that an image named `ubuntu-2204-kube-{{ KUBERNETES_VERSION }}` is available in your environment, corresponding to that Kubernetes version.

**Note:** All clusters created from the dev-test ClusterClass will require a secret named `dev-test-cloud-config` with the `clouds.yaml` to be used by CAPO for interacting with OpenStack.
You can create it manually or see below how to make Tilt automate it.

### Automatically applying kustomizations with Tilt

This explains how to automatically create the secret containing `clouds.yaml`.
The same procedure can be used for any other things you want to create in the cluster.

Ensure the specified path (`/path/to/kustomize/secret/configuration`) contains both the `clouds.yaml` file and a `kustomization.yaml` file. The `kustomization.yaml` should define the necessary resources, such as a Kubernetes secret, using the `clouds.yaml` file.

For example, if you want to automatically create a secret named `dev-test-cloud-config` with the content of your `clouds.yaml` every time you do `tilt up`, you could do the following.
Expand Down Expand Up @@ -144,11 +161,51 @@ secretGenerator:
type: Opaque
```

If you now add `/tmp/capo-dev` to the `additional_kustomizations`, tilt will automatically apply
the secret when you do `tilt up`.
If you now add `/tmp/capo-dev` to the `additional_kustomizations`, tilt will automatically apply
the secret.

To check that the kustomization produces the desired output, do `kustomize build /tmp/capo-dev`.

## Using the 'dev-test' ClusterClass without Tilt

If you want to use the ClusterClass without Tilt, you will need to follow these steps instead of the above.

### Creating a Kind Cluster

Create a Kind cluster and deploy CAPO.

**Note:** As the dev-test ClusterClass is made for development, it may be using a newer API version than what is in the latest release.
You *will need* to [use local artifacts](https://cluster-api.sigs.k8s.io/clusterctl/developers#use-local-artifacts) for this to work in most cases!

```bash
kind create cluster
export CLUSTER_TOPOLOGY=true
clusterctl init --infrastructure openstack
```

### Secret Configuration

CAPO needs a clouds.yaml file in order to manage the OpenStack resources needed for the Cluster. This should be supplied as a secret named `dev-test-cloud-config`. You can create this secret for example with:

```bash
kubectl create secret generic dev-test-cloud-config --from-file=clouds.yaml
```

### Apply the ClusterClass and create Clusters

You can use `clusterctl` to render the ClusterClass:

```bash
clusterctl generate yaml --from templates/clusterclass-dev-test.yaml
```

Create a cluster using the development template, that makes use of the ClusterClass:

```bash
clusterctl generate cluster my-cluster --kubernetes-version=v1.29.0 --from templates/cluster-template-development.yaml > my-cluster.yaml
kubectl apply -f my-cluster.yaml
```

## Running E2E tests locally

You can run the E2E tests locally with:
Expand Down Expand Up @@ -445,4 +502,4 @@ This sections goal is to gather various insights into the API design that can se

Starting from v1alpha8 both `OpenStackMachineStatus` and `BastionsStatus` feature a field named `referencedResources` which aims to include fields that list individual IDs of the resources associated with the machine or bastion. These IDs are calculated on machine or bastion creation and are not intended to be changed during the object lifecycle.

Having all the IDs of related resources saved in the statuses allows CAPO to make easy decisions about deleting the related resources when deleting the VM corresponding to the machine or bastion.
Having all the IDs of related resources saved in the statuses allows CAPO to make easy decisions about deleting the related resources when deleting the VM corresponding to the machine or bastion.
61 changes: 46 additions & 15 deletions templates/clusterclass-dev-test.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,17 @@ spec:
ref:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
name: ${CLUSTER_NAME}-control-plane-template
name: dev-test-control-plane
machineInfrastructure:
ref:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
name: ${CLUSTER_NAME}-control-plane-machine-template
name: dev-test-control-plane-machine
infrastructure:
ref:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackClusterTemplate
name: ${CLUSTER_NAME}-openstackcluster-template
name: dev-test-openstackcluster
workers:
machineDeployments:
- class: default-worker
Expand All @@ -26,17 +26,48 @@ spec:
ref:
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
name: ${CLUSTER_NAME}-default-worker-bootstraptemplate
name: dev-test-default-worker-bootstraptemplate
infrastructure:
ref:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
name: ${CLUSTER_NAME}-default-worker-machine-template
name: dev-test-default-worker-machine
patches:
- name: controlPlaneImage
description: "Sets the OpenStack image that is used for creating the servers."
definitions:
- selector:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
matchResources:
controlPlane: true
jsonPatches:
- op: add
path: /spec/template/spec/image/name
valueFrom:
template: |
ubuntu-2204-kube-{{ .builtin.controlPlane.version }}
- name: workerImage
description: "Sets the OpenStack image that is used for creating the servers."
definitions:
- selector:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
matchResources:
machineDeploymentClass:
names:
- default-worker
jsonPatches:
- op: add
path: /spec/template/spec/image/name
valueFrom:
template: |
ubuntu-2204-kube-{{ .builtin.machineDeployment.version }}
---
apiVersion: bootstrap.cluster.x-k8s.io/v1beta1
kind: KubeadmConfigTemplate
metadata:
name: ${CLUSTER_NAME}-default-worker-bootstraptemplate
name: dev-test-default-worker-bootstraptemplate
spec:
template:
spec:
Expand All @@ -51,7 +82,7 @@ spec:
apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: KubeadmControlPlaneTemplate
metadata:
name: ${CLUSTER_NAME}-control-plane-template
name: dev-test-control-plane
spec:
template:
spec:
Expand Down Expand Up @@ -80,7 +111,7 @@ spec:
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackClusterTemplate
metadata:
name: ${CLUSTER_NAME}-openstackcluster-template
name: dev-test-openstackcluster
spec:
template:
spec:
Expand All @@ -91,38 +122,38 @@ spec:
- 8.8.8.8
identityRef:
kind: Secret
name: ${CLUSTER_NAME}-cloud-config
name: dev-test-cloud-config
managedSecurityGroups: true
nodeCidr: 10.6.0.0/24
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
metadata:
name: ${CLUSTER_NAME}-control-plane-machine-template
name: dev-test-control-plane-machine
spec:
template:
spec:
cloudName: ${OPENSTACK_CLOUD:=capo-e2e}
flavor: ${OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR:=m1.medium}
identityRef:
kind: Secret
name: ${CLUSTER_NAME}-cloud-config
name: dev-test-cloud-config
image:
name: ubuntu-2204-kube-${KUBERNETES_VERSION}
name: overridden-by-patch
sshKeyName: ${OPENSTACK_SSH_KEY_NAME:=""}
---
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha8
kind: OpenStackMachineTemplate
metadata:
name: ${CLUSTER_NAME}-default-worker-machine-template
name: dev-test-default-worker-machine
spec:
template:
spec:
cloudName: ${OPENSTACK_CLOUD:=capo-e2e}
flavor: ${OPENSTACK_NODE_MACHINE_FLAVOR:=m1.small}
identityRef:
kind: Secret
name: ${CLUSTER_NAME}-cloud-config
name: dev-test-cloud-config
image:
name: ubuntu-2204-kube-${KUBERNETES_VERSION}
name: overridden-by-patch
sshKeyName: ${OPENSTACK_SSH_KEY_NAME:=""}