Rancher Wins v0.5.0 upgrade endlessly fails?

[bsnuggs1]

Has anyone ran into an issue where the system-upgrade-controller triggers an upgrade of rancher/wins:v0.5.0, but never completes it? I see several pods on each windows node where it's error'd while attempting to apply an upgrade. When I inspect the pods of the rancher/wins upgrade, I see the following:

INFO[2025-05-21T13:07:51-04:00] Building Initial State...                    
FATA[2025-05-21T13:07:51-04:00] could not build initial state for rancher-wins: could not open rancher-wins service while building initial state: rancher-wins service does not exist 

From looking at the jobs that get created, it seems like the job is missing a few details to accomplish the upgrade. Since I can't edit the jobs directly, I cloned it to try and see if I could get it to run to completion..

1. It is missing the correct securityContext to access host services. Adding the following to spec (or to the container) should resolve the issue:

securityContext:
  windowsOptions:
    hostProcess: true
    runAsUserName: "NT AUTHORITY\\Local service"

2. After trying to add the above, it seems there are more issues because now the following issue occurs:

INFO[2025-05-21T13:26:50-04:00] Building Initial State...                    
FATA[2025-05-21T13:26:50-04:00] could not build initial state for rancher-wins: could not open rancher-wins config while building initial state: could not decode config: open c:/etc/rancher/wins/config: Access is denied. 

Which feels like a bug since it would need to access that file from c:/host/etc/rancher/wins/config instead

How can I modify the controller that keeps creating these jobs? Or should I just do a manual upgrade instead?

Originally posted by @bsnuggs1 in rancher/system-upgrade-controller#368

[bsnuggs1]

I was able to do a workaround for this, but it's not ideal. You can ensure that the latest version of wins is installed by going to the "Registration" tab of the Cluster Management page for your cluster. If you run the registration script on the node again (without uninstalling), it will upgrade rancher wins to v0.5.0 for you. NOTE: Make sure you configure the node to have the same options as you did in the previous install (control plane, worker, etc).

However, you will still have the constant job errors from the system-upgrade-controller. This can be worked around as well. Using the information from this ticket here, it's possible to "fake" a completion of an upgrade.

[HarrisonWAffel]

Hi @bsnuggs1 thanks for opening this issue and sharing your workaround. Could you provide some additional information about your environment, such as the version of Rancher you are running, the version of Windows server you are using, and the k8s version of your cluster?

The Windows SUC plan was updated in both 2.9 and 2.10 to deploy as a host process pod, so it is interesting that your job does not have that field set in the security context. If possible, could you share the content of the windows SUC plan in your cluster (with any sensitive information redacted), as that should contain the intended security context configuration for the underlying jobs. It would also be helpful to know if the config file and associated wins directory has any custom ACL's set that may prevent access.

[bsnuggs1]

No problem! I've put the plan from my cluster below. In addition to updating the Job, I also tried updating the Plan directly (after I understood how the upgrade controller worked more) to include the securityContext I referenced above. It gave the same result, but at least I was properly changing the job now.

The only ACLs I'm aware of are the default ones set by using the registration script. I've also added the output from the registration script below.

Plan

apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  annotations:
    meta.helm.sh/release-name: cluster-managed-system-agent
    meta.helm.sh/release-namespace: cattle-system
    objectset.rio.cattle.io/id: default-cluster-managed-system-agent
    upgrade.cattle.io/digest: spec.upgrade.envs,spec.upgrade.envFrom
  labels:
    app.kubernetes.io/managed-by: Helm
    objectset.rio.cattle.io/hash: f7e8df7f2989981142ca0bce497ec6c9dda58cb012decff4
  name: system-agent-upgrader-windows
  namespace: cattle-system
spec:
  concurrency: 10
  nodeSelector:
    matchExpressions:
      - key: kubernetes.io/os
        operator: In
        values:
          - windows
  secrets:
    - name: system-agent-upgrade-generation
  serviceAccountName: system-agent-upgrader
  tolerations:
    - operator: Exists
  upgrade:
    envFrom:
      - secretRef:
          name: stv-aggregation
    envs:
      - name: STRICT_VERIFY
        value: 'false'
    image: rancher/wins
  version: v0.5.0
status:
  conditions:
    - lastUpdateTime: '2025-05-22T15:05:16Z'
      reason: Version
      status: 'True'
      type: LatestResolved
    - lastUpdateTime: '2025-05-22T15:05:16Z'
      reason: PlanIsValid
      status: 'True'
      type: Validated
    - lastUpdateTime: '2025-05-22T03:59:12Z'
      status: 'True'
      type: Complete
  latestHash: 54b01a9ac0225653c4ac6155b784a064377592e60849bdb6f42f36f160a8fe41af7fe2b6
  latestVersion: v0.5.0

Registration Script Output

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32088    0 32088    0     0   122k      0 --:--:-- --:--:-- --:--:--  125k
Set-ExecutionPolicy : Windows PowerShell updated your execution policy successfully, but the setting is overridden by
a policy defined at a more specific scope.  Due to the override, your shell will retain its current effective
execution policy of RemoteSigned. Type "Get-ExecutionPolicy -List" to view your execution policy settings. For more
information please see "Get-Help Set-ExecutionPolicy".
At line:1 char:81
+ ... install.ps1; Set-ExecutionPolicy Bypass -Scope Process -Force; ./inst ...
+                  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : PermissionDenied: (:) [Set-ExecutionPolicy], SecurityException
    + FullyQualifiedErrorId : ExecutionPolicyOverride,Microsoft.PowerShell.Commands.SetExecutionPolicyCommand
INFO: Windows feature: 'Containers' is installed. Installation will proceed.
INFO: Using default agent configuration directory C:/etc/rancher/wins
INFO: Using default agent var directory C:/var/lib/rancher/agent
INFO: Setting restricted ACL on C:\etc\rancher\wins directory
INFO: Setting restricted ACL on C:\var\lib\rancher\agent directory
INFO: Successfully tested Rancher connection.
INFO: Checking if rancher-wins service exists
INFO: rancher-wins service found, stopping now
INFO: Downloading Wins from <REDACTED>
INFO: Successfully downloaded the wins binary.
INFO: Setting restricted ACL on C:\etc\rancher\wins\config
INFO: Generating Cattle ID
INFO: Cattle ID was already detected as <REDACTED>. Not generating a new one.
INFO: Successfully downloaded Rancher connection information.
INFO: Setting restricted ACL on C:\var\lib\rancher\agent\rancher2_connection_info.json
INFO: Checking if rancher-wins service exists.

Status   Name               DisplayName
------   ----               -----------
Stopped  rancher-wins       Rancher Wins
INFO: Starting rancher-wins service.

[HarrisonWAffel]

@bsnuggs1 Thanks for the quick response!

If you do not have any additional ACLs configured, the issue is likely the user that you have specified when manually adding the security context. The SUC plan configured by Rancher is expected to run with NT AUTHORITY\SYSTEM as the runAsUserName value. If you change the user in the security context that issue may resolve itself.

The root issue seems to be that the plan has not been updated with the appropriate security context fields post upgrade of Rancher. This could potentially be an instance of #49221. Could you check the fleet UI in the local Rancher cluster and see if there are any modified bundles there?

The following information would also be helpful, so I can work on reproducing this issue locally:

• Your Rancher version, as well as any versions you are upgrading away from
• How Rancher is deployed (Helm, Docker, etc.)
• The k8s version of the downstream windows cluster
• In the local cluster, what is the version of the fleet agent running?

Thanks!

[bsnuggs1]

Could you provide some additional information about your environment, such as the version of Rancher you are running, the version of Windows server you are using, and the k8s version of your cluster?

Oh whoops, I knew I had forgotten to add something, sorry about that. Here's the details about the environment:

Rancher Version:
2.10.3

Deployment Method:
Helm

Windows Machine Version:
Windows Server 2019

Downstream Windows Cluster Version:
v1.31.8+rke2r1

Fleet Agent Version:
v0.11.4

If you do not have any additional ACLs configured, the issue is likely the user that you have specified when manually adding the security context. The SUC plan configured by Rancher is expected to run with NT AUTHORITY\SYSTEM as the runAsUserName value. If you change the user in the security context that issue may resolve itself.

The plan I posted in the previous comment is before I made any changes to the securityContext. This had resulted in blank context where securityContext={}. When I added the securityContext to the plan, I still received the "Access is Denied" error, so I had reverted the change.

The root issue seems to be that the plan has not been updated with the appropriate security context fields post upgrade of Rancher. This could potentially be an instance of #49221. Could you check the fleet UI in the local Rancher cluster and see if there are any modified bundles there?

I checked the UI and I see there is a modified bundle, and it's also for the cluster that was experiencing the issue. Extracted yaml below:

apiVersion: fleet.cattle.io/v1alpha1
kind: Bundle
metadata:
  annotations:
    objectset.rio.cattle.io/applied: >-
      <REDACTED>
    objectset.rio.cattle.io/id: manage-system-agent
    objectset.rio.cattle.io/owner-gvk: provisioning.cattle.io/v1, Kind=Cluster
    objectset.rio.cattle.io/owner-name: cluster
    objectset.rio.cattle.io/owner-namespace: fleet-default
  labels:
    objectset.rio.cattle.io/hash: <REDACTED>
  name: cluster-managed-system-agent
  namespace: fleet-default
  resourceVersion: '651264475'
  uid: <REDACTED>
spec:
  correctDrift:
    enabled: true
  defaultNamespace: cattle-system
  ignore: {}
  resources:
    - content: >-
        {"kind":"Plan","apiVersion":"upgrade.cattle.io/v1","metadata":{"name":"system-agent-upgrader","namespace":"cattle-system","creationTimestamp":null,"annotations":{"upgrade.cattle.io/digest":"spec.upgrade.envs,spec.upgrade.envFrom"}},"spec":{"concurrency":10,"nodeSelector":{"matchExpressions":[{"key":"kubernetes.io/os","operator":"In","values":["linux"]}]},"serviceAccountName":"system-agent-upgrader","version":"v0.3.11-suc","secrets":[{"name":"stv-aggregation"},{"name":"system-agent-upgrade-generation"}],"tolerations":[{"operator":"Exists"}],"upgrade":{"image":"rancher/system-agent","envs":[{"name":"STRICT_VERIFY","value":"false"}],"envFrom":[{"secretRef":{"name":"stv-aggregation"}}]}},"status":{}}
      name: Plan-cattle-system-system-agent-upgrader-52ca8b50b3e8.yaml
    - content: >-
        {"kind":"Plan","apiVersion":"upgrade.cattle.io/v1","metadata":{"name":"system-agent-upgrader-windows","namespace":"cattle-system","creationTimestamp":null,"annotations":{"upgrade.cattle.io/digest":"spec.upgrade.envs,spec.upgrade.envFrom"}},"spec":{"concurrency":10,"nodeSelector":{"matchExpressions":[{"key":"kubernetes.io/os","operator":"In","values":["windows"]}]},"serviceAccountName":"system-agent-upgrader","version":"v0.5.0","secrets":[{"name":"system-agent-upgrade-generation"}],"tolerations":[{"operator":"Exists"}],"upgrade":{"image":"rancher/wins","envs":[{"name":"STRICT_VERIFY","value":"false"}],"envFrom":[{"secretRef":{"name":"stv-aggregation"}}],"securityContext":{"windowsOptions":{"runAsUserName":"NT
        AUTHORITY\\SYSTEM","hostProcess":true}}}},"status":{}}
      name: Plan-cattle-system-system-agent-upgrader-windows-6cbda4cf87c9.yaml
    - content: >-
        {"kind":"ServiceAccount","apiVersion":"v1","metadata":{"name":"system-agent-upgrader","namespace":"cattle-system","creationTimestamp":null}}
      name: ServiceAccount-cattle-system-system-agent-upgrader-f7e7b0e7f735.yaml
    - content: >-
        {"kind":"ClusterRole","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system-agent-upgrader","creationTimestamp":null},"rules":[{"verbs":["get"],"apiGroups":[""],"resources":["nodes"]}]}
      name: ClusterRole--system-agent-upgrader-cc32c90ba985.yaml
    - content: >-
        {"kind":"ClusterRoleBinding","apiVersion":"rbac.authorization.k8s.io/v1","metadata":{"name":"system-agent-upgrader","creationTimestamp":null},"subjects":[{"kind":"ServiceAccount","name":"system-agent-upgrader","namespace":"cattle-system"}],"roleRef":{"apiGroup":"rbac.authorization.k8s.io","kind":"ClusterRole","name":"system-agent-upgrader"}}
      name: ClusterRoleBinding--system-agent-upgrader-0582ce53a1f0.yaml
    - content: >-
        {"kind":"Secret","apiVersion":"v1","metadata":{"name":"system-agent-upgrade-generation","namespace":"cattle-system","creationTimestamp":null},"stringData":{"cluster-uid":"<REDACTED>","generation":"1"}}
      name: Secret-cattle-system-system-agent-upgrade-generation-af25-819fc.yaml
  targets:
    - clusterName: cluster
      clusterSelector:
        matchExpressions:
          - key: provisioning.cattle.io/unmanaged-system-agent
            operator: DoesNotExist
      ignore: {}
status:
  conditions:
    - lastUpdateTime: '2025-05-22T03:11:25Z'
      message: >-
        Modified(1) [Cluster fleet-default/cluster]; plan.upgrade.cattle.io
        cattle-system/system-agent-upgrader-windows modified
        {"spec":{"upgrade":{"securityContext":{"windowsOptions":{"hostProcess":true,"runAsUserName":"NT
        AUTHORITY\\SYSTEM"}}}}}
      status: 'False'
      type: Ready
    - lastUpdateTime: '2025-05-17T01:37:25Z'
      status: 'True'
      type: Processed
  display:
    readyClusters: 0/1
    state: Modified
  maxNew: 50
  maxUnavailable: 1
  maxUnavailablePartitions: 0
  observedGeneration: 9
  partitions:
    - count: 1
      maxUnavailable: 1
      name: All
      summary:
        desiredReady: 1
        modified: 1
        nonReadyResources:
          - bundleState: Modified
            modifiedStatus:
              - apiVersion: upgrade.cattle.io/v1
                kind: Plan
                name: system-agent-upgrader-windows
                namespace: cattle-system
                patch: >-
                  {"spec":{"upgrade":{"securityContext":{"windowsOptions":{"hostProcess":true,"runAsUserName":"NT
                  AUTHORITY\\SYSTEM"}}}}}
            name: fleet-default/cluster
        ready: 0
  resourceKey:
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRole
      name: system-agent-upgrader
    - apiVersion: rbac.authorization.k8s.io/v1
      kind: ClusterRoleBinding
      name: system-agent-upgrader
    - apiVersion: upgrade.cattle.io/v1
      kind: Plan
      name: system-agent-upgrader
      namespace: cattle-system
    - apiVersion: upgrade.cattle.io/v1
      kind: Plan
      name: system-agent-upgrader-windows
      namespace: cattle-system
    - apiVersion: v1
      kind: Secret
      name: system-agent-upgrade-generation
      namespace: cattle-system
    - apiVersion: v1
      kind: ServiceAccount
      name: system-agent-upgrader
      namespace: cattle-system
  resourcesSha256Sum: c7b55e2a3fd6cf78f558e931b6525f7fbca2e7b35c73faf8818670294c8f67bd
  summary:
    desiredReady: 1
    modified: 1
    nonReadyResources:
      - bundleState: Modified
        modifiedStatus:
          - apiVersion: upgrade.cattle.io/v1
            kind: Plan
            name: system-agent-upgrader-windows
            namespace: cattle-system
            patch: >-
              {"spec":{"upgrade":{"securityContext":{"windowsOptions":{"hostProcess":true,"runAsUserName":"NT
              AUTHORITY\\SYSTEM"}}}}}
        name: fleet-default/cluster
    ready: 0
  unavailable: 0
  unavailablePartitions: 0

[HarrisonWAffel]

Thanks, the content of that bundle confirms that it is another instance of #49221.

I've had difficulty reproducing this issue internally, and we did not observe it when testing the PR which introduced the upgrade process itself. I've reattempted the upgrade from 2.9.3 to 2.10.3 today after provisioning a 1 linux + 1 windows node cluster and so far have not had luck reproducing this issue (Rancher was deployed using Helm on rke2 1.30.4). I'm going to try again with a beefier Rancher server to see if that has any influence on reproducibility.

In your case, you should be able to delete the Windows SUC plan in the downstream cluster explorer UI (under the 'More Resources' and 'Upgrade Controller' section) to resolve the modified bundle. Rancher will automatically recreate the bundle and properly populate the security context the next time that the cluster reconciles. At this point the upgrade of rancher-wins should occur, and you won't see a modified fleet bundle when performing future upgrades. If you prefer, you could also reattempt your workaround while setting the runAsUser to NT AUTHORITY\\SYSTEM instead of NT AUTHORITY\\Local service.

[bsnuggs1]

@HarrisonWAffel Thanks for clarifying! The good news is that this issue is resolved for now and I can use the fix you proposed instead if I encounter it again, as I do have another rancher instance to upgrade later on this year.

I'm not sure if this will help, but one important fact about the upgrade I've done is that its from a series of upgrades. Unfortuantely, I wasn't necessarily upgrading rancher as often as I should 😅

This issue occurred after upgrading rancher all the way from 2.7.10 to 2.10.3. And the clusters from 1.24 to 1.31.8. This was done procedurally where the clusters are upgraded to the maximum version supported by rancher step by step (1.24>1.25>12.6, etc.), then rancher is upgraded to the next major version. It could be possible that the issue is introduced via some residue from a much earlier version?

[HarrisonWAffel]

That's super useful to know, thanks for calling that out! I'll take some time and follow a similar upgrade path to see if I can reproduce and root cause the issue. Glad to hear your environment is in a good place in the mean time, thanks for all the detailed feedback in this issue!

[bsnuggs1]

No problem, I hope the feedback helps!

I'll follow up when I get a chance to do the other instance upgrade. For now, I'll close this issue.