gpu-device-plugin does not respect tolerations

[audiophonicz]

Describe the support request
I just updated NFD and GPU Plugins from 0.27 to 0.31.1 due to K3s version increment.

Installing gpu-device-plugin via Helm chart per INSTALL.md

NFD Node-Feature-Discovery tolerations work. Apparently Device-Plugins-Operator does not support tolerations, but thats OK cuz it can run on any of the other 5 nodes.

GPU-Device-Plugin Chart Values.yaml shows tolerations as a supported field.

Helm accepts both --set commands and values.yaml inputs for Tolerations, but the Daemonset does not respect them.

> helm install gpu-device-plugin intel/intel-device-plugins-gpu \
>   --namespace inteldeviceplugins-system --version 0.31.1  -f gpu-device-plugin-values.yml

W1207 23:08:44.767727 3079715 warnings.go:70] unknown field "spec.tolerations"

Helm Get Values Output Showing Tolerations

>helm get values gpu-device-plugin -n inteldeviceplugins-system
USER-SUPPLIED VALUES:
allocationPolicy: none
enableMonitoring: true
image:
  hub: intel
  tag: ""
initImage:
  enable: false
  hub: intel
  tag: ""
logLevel: 2
name: worker
nodeFeatureRule: true
nodeSelector:
  intel.feature.node.kubernetes.io/gpu: "true"
resourceManager: false
sharedDevNum: 10
tolerations:
- effect: NoSchedule
  key: dedicated
  operator: Equal
  value: transcode

But the pod/daemonset does not accept tolerations and throws them away. kubectl edit directly will save as a valid edit but the tolerations will be thrown away immediately.

GPU-Device-Plugin Daemonset Manifest

> k get daemonset -n inteldeviceplugins-system intel-gpu-plugin-worker -o yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  annotations:
    deprecated.daemonset.template.generation: "2"
  creationTimestamp: "2024-12-08T04:20:14Z"
  generation: 2
  labels:
    app: intel-gpu-plugin
  name: intel-gpu-plugin-worker
  namespace: inteldeviceplugins-system
  ownerReferences:
  - apiVersion: deviceplugin.intel.com/v1
    blockOwnerDeletion: true
    controller: true
    kind: GpuDevicePlugin
    name: worker
    uid: 6d565661-c726-4359-8ee7-e5a35bee391d
  resourceVersion: "313888847"
  uid: c822c290-bfe7-4388-a5d9-2270ecdc4a2d
spec:
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: intel-gpu-plugin
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: intel-gpu-plugin
    spec:
      containers:
      - args:
        - -v
        - "2"
        - -enable-monitoring
        - -shared-dev-num
        - "10"
        - -allocation-policy
        - none
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: spec.nodeName
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        image: intel/intel-gpu-plugin:0.31.1
        imagePullPolicy: IfNotPresent
        name: intel-gpu-plugin
        resources:
          limits:
            cpu: 100m
            memory: 90Mi
          requests:
            cpu: 40m
            memory: 45Mi
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          seLinuxOptions:
            type: container_device_plugin_t
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /dev/dri
          name: devfs
          readOnly: true
        - mountPath: /sys/class/drm
          name: sysfsdrm
          readOnly: true
        - mountPath: /var/lib/kubelet/device-plugins
          name: kubeletsockets
        - mountPath: /var/run/cdi
          name: cdipath
      dnsPolicy: ClusterFirst
      nodeSelector:
        intel.feature.node.kubernetes.io/gpu: "true"
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: default
      serviceAccountName: default
      terminationGracePeriodSeconds: 30
      volumes:
      - hostPath:
          path: /dev/dri
          type: ""
        name: devfs
      - hostPath:
          path: /sys/class/drm
          type: ""
        name: sysfsdrm
      - hostPath:
          path: /var/lib/kubelet/device-plugins
          type: ""
        name: kubeletsockets
      - hostPath:
          path: /var/run/cdi
          type: DirectoryOrCreate
        name: cdipath
  updateStrategy:
    rollingUpdate:
      maxSurge: 0
      maxUnavailable: 1
    type: RollingUpdate
status:
  currentNumberScheduled: 5
  desiredNumberScheduled: 5
  numberAvailable: 5
  numberMisscheduled: 0
  numberReady: 5
  observedGeneration: 2
  updatedNumberScheduled: 5

Expected Behavior
Tolerations are accepted and daemonset workers are scheduled on all nodes regardless of taint.

System (please complete the following information if applicable):

• OS version: Debian Bookworm 12.7
• Kubernetes: K3s v1.30.6+k3s1
• Kernel version: Linux 6.10.11+bpo-amd64
• Device plugins version: v0.31.1
• Hardware info: N/A - Multi

[tkatila]

Hi @audiophonicz, can you check if the GPU CRD includes the tolerations portion? You mentioned that you upgraded from 0.27 to 0.31. Helm doesn't automatically upgrade the CRDs so they could still be from 0.27.

As the Daemonset is installed via the operator, manual edits to the Daemonset will get dropped by the operator.

[audiophonicz]

Hi @audiophonicz, can you check if the GPU CRD includes the tolerations portion? You mentioned that you upgraded from 0.27 to 0.31. Helm doesn't automatically upgrade the CRDs so they could still be from 0.27.

Good call, my GPU CRD did not update even tho I did uninstall and reinstall both Operator and GPU plugin. I just upgraded from v0.9.2 to v0.16.1. It was v0.16.1 that I was seeing that tolerations are listed for the GPU Plugin CRD in addition to the previously mentioned values files. Unfortunately, no joy.

              tolerations:
                description: Specialized nodes (e.g., with accelerators) can be Tainted
                  to make sure unwanted pods are not scheduled on them. Tolerations
                  can be set for the plugin pod to neutralize the Taint.
                items:
                  description: |-
                    The pod this Toleration is attached to tolerates any taint that matches
                    the triple <key,value,effect> using the matching operator <operator>.
                  properties:
                    effect:
                      description: |-
                        Effect indicates the taint effect to match. Empty means match all taint effects.
                        When specified, allowed values are NoSchedule, PreferNoSchedule and NoExecute.
                      type: string
                    key:
                      description: |-
                        Key is the taint key that the toleration applies to. Empty means match all taint keys.
                        If the key is empty, operator must be Exists; this combination means to match all values and all keys.
                      type: string
                    operator:
                      description: |-
                        Operator represents a key's relationship to the value.
                        Valid operators are Exists and Equal. Defaults to Equal.
                        Exists is equivalent to wildcard for value, so that a pod can
                        tolerate all taints of a particular category.
                      type: string

As the Daemonset is installed via the operator, manual edits to the Daemonset will get dropped by the operator.

This must be my problem then, as my manual edits to the Daemonset are dropped even after being accepted. So how does one add the tolerations to the gpu plugin Daemonset?

I see that in 0.30.0 release says it "adds toleration support to xDevicePlugin CRDs." and I see you worked on 1617 and 1686, but I still do not see a provision for tolerations in the Operator Chart or values.yaml for 0.31.1, let alone to set tolerations for generated Daemonsets. Attempting to add the toleration to the helm command for either is ignored.

Further the deployment for the Operator has no values for tolerations, but the running pod has

tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 30
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 30

so Im not sure where these are even coming from.

[eero-t]

Good call, my GPU CRD did not update even tho I did uninstall and reinstall both Operator and GPU plugin.

Helm intentionally [1] supports neither upgrading nor removing CRDs, so even removing & re-installing the Helm chart does not help, you need to remove CRDs manually before upgrade or (re-install).

This must be my problem then, as my manual edits to the Daemonset are dropped even after being accepted. So how does one add the tolerations to the gpu plugin Daemonset?

Looking at the chart, you should specify toleration in the values.yaml:
https://github.com/intel/helm-charts/blob/intel-device-plugins-gpu-0.31.1/charts/gpu-device-plugin/values.yaml#L21

That will then include it to CRD installed by Helm:
https://github.com/intel/helm-charts/blob/intel-device-plugins-gpu-0.31.1/charts/gpu-device-plugin/templates/gpu.yaml#L23

Add operator should then apply it from CRD to the Deployment (I assume, as I'm not developer for this).

To get CRDs to be updated, you need to remove existing ones manually before doing upgrade though!

That procedure should be documented (with examples) in the chart README, like the other Helm charts [2] installing CRDs do => @tkatila ?

---

[1] Helm & CRDs:

• Summary: https://helm.sh/docs/chart_best_practices/custom_resource_definitions/
• Problem details: https://github.com/helm/community/blob/main/hips/hip-0011.md

[2] Intel DRA GPU driver Helm chart mentions the issue, but e.g. Prometheus Helm chart lists also needed commands: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/README.md#uninstall-helm-chart

[tkatila]

I verified that setting tolerations to the GPU CR does provide them to the GPU plugin daemonset.

First I installed the operator:

helm install dp-operator intel/intel-device-plugins-operator --namespace inteldeviceplugins-system --create-namespace --version 0.31.1

Then the GPU plugin:

helm install gpu intel/intel-device-plugins-gpu --namespace inteldeviceplugins-system --create-namespace --set nodeFeatureRule=true --set tolerations[0].effect=NoSchedule --set tolerations[0].key=foo.bar\/dipdap --set tolerations[0].operator=Equal

The resulting daemonset has the added toleration:

kubectl get pod -n inteldeviceplugins-system intel-gpu-plugin-gpudeviceplugin-sample-76zlh -o yaml | yq .spec.tolerations[0]
{
  "effect": "NoSchedule",
  "key": "foo.bar/dipdap",
  "operator": "Equal"
}

Looking at your initial post, I think your problem is that you tried to set "spec.tolerations" as the correct key is just "tolerations"?

[tkatila]

To get CRDs to be updated, you need to remove existing ones manually before doing upgrade though!

I hadn't considered that uninstalling the chart wouldn't also remove the CRDs. That's annoying.

That procedure should be documented (with examples) in the chart README, like the other Helm charts [2] installing CRDs do => @tkatila ?

Yep, I think we need to add uninstall steps here as well as to the helm chart documentation.

[audiophonicz]

@tkatila thanks for confirming yours worked and letting me know the CRDs needed updated. After updating CRDs, I had to uninstall the gpu-device-plugin and then reinstall. Upgrading with added values did not push it out to the pods, but the fresh install did. I also see it in the daemonset now.

Thank you for your work and your help.