Support for per-zone PDB (#253)

* Support for per-zone PDB

Includes;

* validating webhook configuration to handle pod evictions
* validating webhook to validate zpdb configurations
* custom resource definition for zpdb configurations
* eviction handler for enforcing a zone aware and partition aware pdb

Co-authored-by: Andy Asp <[email protected]>

Author: tcp13equals2

CHANGELOG.md

@@ -16,6 +16,7 @@
   * `k8s.io/apimachinery` from `v0.33.1` to `v0.33.3`
   * `k8s.io/client-go` from `v0.33.1` to `v0.33.3`
 * [ENHANCEMENT] Automatically patch new validating and mutating rollout-operator webhooks with the self-signed CA if they are created after rollout-operator starts. #262
+* [ENHANCEMENT] Support for zone and partition aware pod disruption budgets, enabling finer control over pod eviction policies. #253
 * [BUGFIX] Always configure HTTP client with a timeout. #240
 * [BUGFIX] Use a StatefulSet's `.spec.serviceName` when constructing the delayed downscale endpoint for a pod. #258
 

README.md

@@ -132,6 +132,15 @@ Prometheus metrics endpoint.
 
 Offers a `ValidatingAdmissionWebhook` that rejects the requests that decrease the number of replicas in objects labeled as `grafana.com/no-downscale: true`. See [Webhooks](#webhooks) section below. 
 
+#### `/pods/eviction`
+
+Offers a `ValidatingAdmissionWebhook` which can apply a `ZoneAwarePodDisruptionBudget` to administer voluntary pod evictions. See [ZoneAwarePodDisruptionBudget](#zoneawarepoddisruptionbudget-zpdb) section below.
+
+#### `/admission/zpdb-validation`
+
+Offers a `ValidatingAdmissionWebhook` to validate `ZoneAwarePodDisruptionBudget` configuration files and will reject any misconfigured files.
+
+
 ### RBAC
 
 When running the `rollout-operator` as a pod, it needs a Role with at least the following privileges:
@@ -163,6 +172,14 @@ rules:
   - statefulsets/status
   verbs:
   - update
+- apiGroups:
+  - rollout-operator.grafana.com
+  resources:
+  - zoneawarepoddisruptionbudgets
+  verbs:
+  - get
+  - list
+  - watch
 ```
 
 (Please see [Webhooks](#webhooks) section below for extra roles required when using the HTTPS server for webhooks.)
@@ -415,3 +432,95 @@ subjects:
 
 Whenever the certificate expires, the `rollout-operator` will detect it and will restart, which will trigger the self-signed certificate generation again if it's configured.
 The default expiration for the self-signed certificate is 1 year and it can be changed by setting the flag `-server-tls.self-signed-cert.expiration`.
+
+# ZoneAwarePodDisruptionBudget (ZPDB)
+
+A custom `PodDisruptionBudget` is available for use with the `rollout-operator`. 
+
+This is for use with `StatefulSets` which span multiple logical zones and allows for the budget to be evaluated against the pods in other zones.
+
+Unlike a regular `PodDisruptionBudget` which evaluates across all pods, the `ZoneAwarePodDisruptionBudget` evaluates against unavailable pod counts within a zone, and only allows an eviction if no other zone is disrupted.
+
+This allows an operator to perform maintenance on a single zone whilst ensuring sufficient pod availability in other zones.
+
+Consider the following topology where the `ZPDB` has `maxUnavailable` set to 1:
+
+* StatefulSet `ingester-zone-a` manages pods `ingester-zone-a-0` and `ingester-zone-a-1`
+* StatefulSet `ingester-zone-b` manages pods `ingester-zone-b-0` and `ingester-zone-b-1`
+* StatefulSet `ingester-zone-c` manages pods `ingester-zone-c-0` and `ingester-zone-c-1`
+
+When a pod eviction request is received, the availability of the pods in the other zones are considered, as well as the availability in the zone of the pod being evicted.
+
+If `ingester-zone-a-0` is to be evicted, it will be allowed if there are no disruptions in either zone `b` or zone `c`.
+
+If `ingester-zone-a-1` has failed and `ingester-zone-a-0` is to be evicted, this will not be allowed since `maxUnavailable` of 1 is only allowed within this zone.
+
+If `maxUnavailable` is 2, `ingester-zone-a-0` eviction would be granted since zone `a` can have 2 unavailable nodes, and there are no disruptions in zone `b` or zone `c`.
+
+If `ingester-zone-a-0` is to be evicted, and `ingester-zone-b-0` has failed, the eviction request will be denied regardless of the value of `maxUnavailable` because another zone is already disrupted.
+
+*A pod eviction is only allowed if the number of unavailable pods is within the maximum unavailability threshold for the zone and no other zone has a disruption.*
+
+## Partition awareness
+
+The `ZPDB` can be configured for partition awareness. This is intended for workloads like Mimir's ingesters running with ingest storage, where we can tolerate some unavailability in different zones, provided each partition has sufficient availability.
+
+In this configuration, the `ZPDB` determines the partition for a pod being evicted, and considers this eviction against the unavailable counts for ALL pods which serve this partition.
+
+*A pod eviction is only allowed if the number of unavailable pods serving a specific partition is less than the `maxUnavailable` value.*
+
+Using the same topology as the previous section where the `ZPDB` has `maxUnavailable=1`;
+
+If `ingester-zone-b-0` has failed and `ingester-zone-a-1` is to be evicted, it will be allowed as there are no disruptions in either zone `b` or zone `c` for partition `1`.
+
+If `ingester-zone-b-0` has failed and `ingester-zone-a-0` is to be evicted, it will be denied as the partition `0` in zone `b` is disrupted.
+
+## Operations
+
+### Setup
+
+The `ZoneAwarePodDisruptionBudget` is provided as a custom resource.
+
+A pod eviction webhook is registered for approving voluntary pod eviction requests, and a validating webhook is registered for validating `ZPDB` objects.
+
+The following is required to enable the `ZoneAwarePodDisruptionBudget`;
+
+* a custom resource definition for the `ZoneAwarePodDisruptionBudget` kind - a sample is provided in [development](./development/zone-aware-pod-disruption-budget-custom-resource-definition.yaml)
+* a `ValidatingWebhookConfiguration` for registering the `rollout-operator` for pod evictions - a sample is provided in [development](./development/eviction-webhook.yaml)
+* a `ZoneAwarePodDisruptionBudget` kind for each set of `StatefulSets` - see below
+
+Example `ZoneAwarePodDisruptionBudget`;
+
+```yaml
+apiVersion: rollout-operator.grafana.com/v1
+kind: ZoneAwarePodDisruptionBudget
+metadata:
+  name: ingester-rollout
+  namespace: namespace
+  labels:
+    name: ingester-rollout
+spec:
+  maxUnavailable: 1
+  selector:
+    matchLabels:
+      rollout-group: ingester
+  # podNamePartitionRegex: "[a-z\\-]+-zone-[a-z]-([0-9]+)"
+  # podNameRegexGroup: 1
+```
+
+### Configuration options
+
+The exact resource attributes should be referenced via the provided custom resource definition file (see above).
+
+Functionality includes the ability to;
+
+* set a fixed max unavailable pod threshold
+* set the unavailable pod threshold as a percentage. This can only be used in classic zones and can not be used with partition awareness. The percentage is calculated against the StatefulSet's `spec.Replica` count. 
+* set the selector to match the applicable Pods and StatefulSets
+* set the regular expression to determine a partition name from a pod name (if using partition awareness)
+
+Note - `maxUnavailable` can be set to 0. In this case no voluntary evictions in any zone will be allowed.
+
+Note - a validating webhook configuration is provided in [development](./development/zone-aware-pod-disruption-budget-validating-webhook.yaml) which allows the `rollout-operator` to verify a `ZoneAwarePodDisruptionBudget` configuration being created or updated. This will ensure that no invalid configuration can be applied.
+
+Note - the `podNameRegexGroup` allows for the capture group index to be set. This is required if the partition regex has more than one set of groupings `(...)` in the expression. 1-based indexing is used, such that 1 will match the first parenthesized capture group.

cmd/rollout-operator/main.go

@@ -37,6 +37,7 @@ import (
 	"github.com/grafana/rollout-operator/pkg/controller"
 	"github.com/grafana/rollout-operator/pkg/instrumentation"
 	"github.com/grafana/rollout-operator/pkg/tlscert"
+	"github.com/grafana/rollout-operator/pkg/zpdb"
 )
 
 const defaultServerSelfSignedCertExpiration = model.Duration(365 * 24 * time.Hour)
@@ -206,19 +207,23 @@ func main() {
 	dynamicClient, err := dynamic.NewForConfigAndClient(kubeConfig, httpClient)
 	check(errors.Wrap(err, "failed to init dynamicClient"))
 
+	// watches for validating webhooks being added - this is only started if the TLS server is started
 	webhookObserver := tlscert.NewWebhookObserver(kubeClient, cfg.kubeNamespace, logger)
 
-	// Start TLS server if enabled.
-	maybeStartTLSServer(cfg, httpRT, logger, kubeClient, restart, metrics, webhookObserver)
+	// controller for pod eviction - this is only started if the TLS server is started
+	evictionController := zpdb.NewEvictionController(kubeClient, dynamicClient, cfg.kubeNamespace, logger)
 
-	// Init the controller.
+	maybeStartTLSServer(cfg, httpRT, logger, kubeClient, restart, metrics, evictionController, webhookObserver)
+
+	// Init the controller
 	c := controller.NewRolloutController(kubeClient, restMapper, scaleClient, dynamicClient, cfg.kubeNamespace, httpClient, cfg.reconcileInterval, reg, logger)
 	check(errors.Wrap(c.Init(), "failed to init controller"))
 
 	// Listen to sigterm, as well as for restart (like for certificate renewal).
 	go func() {
 		waitForSignalOrRestart(logger, restart)
 		c.Stop()
+		evictionController.Stop()
 		webhookObserver.Stop()
 	}()
 
@@ -240,7 +245,7 @@ func waitForSignalOrRestart(logger log.Logger, restart chan string) {
 	}
 }
 
-func maybeStartTLSServer(cfg config, rt http.RoundTripper, logger log.Logger, kubeClient *kubernetes.Clientset, restart chan string, metrics *metrics, vwo *tlscert.WebhookObserver) {
+func maybeStartTLSServer(cfg config, rt http.RoundTripper, logger log.Logger, kubeClient *kubernetes.Clientset, restart chan string, metrics *metrics, evictionController *zpdb.EvictionController, webhookObserver *tlscert.WebhookObserver) {
 	if !cfg.serverTLSEnabled {
 		level.Info(logger).Log("msg", "tls server is not enabled")
 		return
@@ -281,18 +286,30 @@ func maybeStartTLSServer(cfg config, rt http.RoundTripper, logger log.Logger, ku
 		}
 
 		// Start monitoring for validating webhook configurations and patch if required
-		check(vwo.Init(webHookListener))
-
+		check(webhookObserver.Init(webHookListener))
 	}
 
+	// Start monitoring for zpdb configurations and pods
+	check(evictionController.Start())
+
 	prepDownscaleAdmitFunc := func(ctx context.Context, logger log.Logger, ar v1.AdmissionReview, api *kubernetes.Clientset) *v1.AdmissionResponse {
 		return admission.PrepareDownscale(ctx, rt, logger, ar, api, cfg.useZoneTracker, cfg.zoneTrackerConfigMapName)
 	}
 
+	podEvictionFunc := func(ctx context.Context, _ log.Logger, ar v1.AdmissionReview, _ *kubernetes.Clientset) *v1.AdmissionResponse {
+		return evictionController.HandlePodEvictionRequest(ctx, ar)
+	}
+
+	zpdbValidationFunc := func(ctx context.Context, l log.Logger, ar v1.AdmissionReview, _ *kubernetes.Clientset) *v1.AdmissionResponse {
+		return admission.ZoneAwarePdbValidatingWebhookHandler(ctx, l, ar)
+	}
+
 	tlsSrv, err := newTLSServer(cfg, logger, cert, metrics)
 	check(errors.Wrap(err, "failed to create tls server"))
 	tlsSrv.Handle(admission.NoDownscaleWebhookPath, admission.Serve(admission.NoDownscale, logger, kubeClient))
 	tlsSrv.Handle(admission.PrepareDownscaleWebhookPath, admission.Serve(prepDownscaleAdmitFunc, logger, kubeClient))
+	tlsSrv.Handle(zpdb.PodEvictionWebhookPath, admission.Serve(podEvictionFunc, logger, kubeClient))
+	tlsSrv.Handle(admission.ZpdbValidatorWebhookPath, admission.Serve(zpdbValidationFunc, logger, kubeClient))
 	check(errors.Wrap(tlsSrv.Start(), "failed to start tls server"))
 }
 

development/README.md

@@ -1,13 +1,126 @@
-This directory contains Kubernetes manifests to start an instance of the rollout-operator locally.
+# Quick Start
+
+This directory contains Kubernetes manifests to start an instance of the `rollout-operator` locally.
 
 To use it:
 
-* Build the rollout-operator image: `make build-image`
+* Build the `rollout-operator` image: `make build-image`
 * Make the image available to your Kubernetes cluster (not required for use with Docker Desktop)
 * Apply the Kubernetes manifests: `./apply.sh`
-* Port forward to the operator service: `kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8080:80`
+* Port forward to the operator service;
+```
+kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8080:80
+kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8443:443
+```
 * Port forward to the Jaeger UI: `kubectl --namespace=rollout-operator-development port-forward svc/jaeger 16686:16686`
 
-You'll then be able to access the rollout operator at `http://localhost:8080`, and the Jaeger tracing UI at `http://localhost:16686`.
+You'll then be able to access the rollout operator at `http://localhost:8080`, access the rollout operator webhooks at `http://localhost:8443` and the Jaeger tracing UI at `http://localhost:16686`.
+
+You can use the StatefulSets to exercise the operator across a multi-zone `test-app` environment.
+
+# ZoneAwarePodDisruptionBudget (ZPDB)
+
+Included is a `ZoneAwarePodDisruptionBudget` which can be used to enforce a multi-zone pod disruption budget.
+
+By default, this will be applied to the `test-app` Pods and StatefulSets.
+
+To disable this functionality from the `test-app`;
+
+```text
+kubectl delete -f rollout-operator-zone-aware-pod-disruption-budget.yaml
+```
+
+# Minikube & Docker Desktop
+
+Note - if you are using local `Docker Desktop` and `minikube` and you intend to use locally built images, ensure that you are using Minikube's Docker Daemon so any images you build will be available inside the Minikube cluster.
+
+Additionally, ensure that you set the container image reference to `imagePullPolicy: Never`.
+
+```
+cd ~/rollout-operator
+minikube start
+eval $(minikube docker-env)
+make build-image
+
+(
+    cd development
+    ./apply.sh
+)
+```
+
+# Useful commands
+
+The following are useful commands when running tests with the `rollout-operator` and the `ZoneAwarePodDisruptionBudget`
+
+List custom resource definitions;
+```
+kubectl get crds -n rollout-operator-development
+```
+
+List custom resources;
+```
+kubectl get zoneawarepoddisruptionbudgets -n rollout-operator-development
+```
+
+List the custom resource configuration by name;
+```
+kubectl get zoneawarepoddisruptionbudget test-app -n rollout-operator-development -o yaml 
+```
+
+Port forward to the rollout-operator;
+```
+kubectl --namespace=rollout-operator-development port-forward svc/rollout-operator 8443:443
+```
+
+Tail logs for the rollout-operator;
+```
+kubectl logs -f `kubectl get pods -n rollout-operator-development | grep rollout-operator | awk '{print $1}'` -n rollout-operator-development
+```
+
+Test the pod eviction and `ZPDB`;
+```
+# watch the pod status
+while true; do kubectl get pods -n rollout-operator-development; sleep 1; clear; done
+```
+
+```
+# in another shell - issue a drain for all pods
+kubectl drain --pod-selector rollout-group=test-app minikube &; sleep 5; kubectl uncordon minikube
+```
+
+Note - in the above example the `uncordon` is important as without this the drained pods in the first zone will not be re-deployed. Until these pods are running again the next zone's pods will not be evicted.
+
+This is a limitation of running multiple zones within a single kubernetes node. In a usual deployment each zone would have its pods distributed across different nodes.
+
+Apply an updated `ZPDB`
+```
+# make changes in rollout-operator-zone-aware-pod-disruption-budget.yaml
+kubectl apply -f rollout-operator-zone-aware-pod-disruption-budget.yaml
+```
 
-You can use the `test-app` StatefulSet to exercise the operator.
+Test the pod eviction webhook manually;
+```
+curl --insecure -X POST "https://127.0.0.1:8443/admission/pod-eviction" \
+  -H "Content-Type: application/json" \
+  --data '{
+    "apiVersion": "admission.k8s.io/v1",
+    "kind": "AdmissionReview",
+    "request": {
+      "uid": "test-eviction-123",
+      "kind": {"group": "policy", "version": "v1", "kind": "Eviction"},
+      "resource": {"group": "policy", "version": "v1", "resource": "evictions"},
+      "name": "test-app-zone-a-0",
+      "namespace": "rollout-operator-development",
+      "operation": "CREATE",
+      "subResource": "eviction",
+      "userInfo": {"username": "test-user"},
+      "dryRun": false,
+      "object": {
+        "apiVersion": "policy/v1",
+        "kind": "Eviction",
+        "metadata": {"name": "test-app-zone-a-0", "namespace": "rollout-operator-development"},
+        "deleteOptions": {"gracePeriodSeconds": 30}
+      }
+    }
+  }'
+```

development/apply.sh

@@ -19,4 +19,5 @@ select yn in "Yes" "No"; do
 done
 
 kubectl apply --wait -f "$SCRIPT_DIR/namespace.yaml"
-find "$SCRIPT_DIR" -type f -name '*.yaml' -not -name 'namespace.yaml' -exec kubectl apply --namespace=rollout-operator-development --wait -f {} \;
+kubectl apply --wait -f "$SCRIPT_DIR/zone-aware-pod-disruption-budget-custom-resource-definition.yaml"
+find "$SCRIPT_DIR" -type f -name '*.yaml' -not -name 'namespace.yaml' -not -name 'zone-aware-pod-disruption-budget-custom-resource-definition.yaml' -exec kubectl apply --namespace=rollout-operator-development --wait -f {} \;

development/eviction-webhook.yaml

@@ -0,0 +1,33 @@
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: pod-eviction-rollout-operator-development
+  labels:
+    grafana.com/inject-rollout-operator-ca: "true"
+    grafana.com/namespace: rollout-operator-development
+webhooks:
+  - name: pod-eviction-rollout-operator-development.grafana.com
+    clientConfig:
+      service:
+        namespace: rollout-operator-development
+        name: rollout-operator
+        path: /admission/pod-eviction
+        port: 443
+    rules:
+      - operations:
+          - CREATE
+        apiGroups:
+          -
+        apiVersions:
+          - v1
+        resources:
+          - pods/eviction
+        scope: Namespaced
+
+    admissionReviewVersions:
+      - v1
+    namespaceSelector:
+      matchLabels:
+        kubernetes.io/metadata.name: rollout-operator-development
+    failurePolicy: Fail
+    sideEffects: None

development/rollout-operator-role.yaml

@@ -28,3 +28,12 @@ rules:
       - statefulsets/status
     verbs:
       - update
+  - apiGroups:
+      - rollout-operator.grafana.com
+    resources:
+      - zoneawarepoddisruptionbudgets
+    verbs:
+      - get
+      - list
+      - watch
+