klipper-helm doesn't set HOME env var and tries to write on /

[theodiem]

from the helm-install pod:

+ [[ false == \t\r\u\e ]]
+ [[ false == \t\r\u\e ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/kubernetes-replicator.tgz.base64
+ CHART_PATH=/tmp/kubernetes-replicator.tgz
+ [[ ! -f /chart/kubernetes-replicator.tgz.base64 ]]
+ return
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ kubernetes-replicator/kubernetes-replicator == stable/* ]]
+ [[ -n https://helm.mittwald.de ]]
+ [[ -f /auth/username ]]
+ [[ -f /auth/tls.crt ]]
+ helm repo add kubernetes-replicator https://myhelm.de
Error: mkdir /.config: read-only file system

Using rancher/klipper-helm:v0.9.6-build20250616 and k3s 1.32

[brandond]

Are you for some reason using a custom deployment of helm-controller, or have you in some way modified the helm job spec or job image?

[brandond]

The helm job image runs as user klipper-helm, we shouldn't need to manually set the home dir as a this should come from the passwd entry.

Ref: https://github.com/k3s-io/klipper-helm/blob/404c4497a5746f72101de591b3fd2f7603518299/Dockerfile#L32-L41

Its possible something changed in alpine 1.22 but that would be a surprising regression.

[brandond]

If helm now requires $HOME to be set in order to resolve the user's home directory, that is a bug in Helm. The image itself is fine.

brandond@dev01:~$ docker run --rm -it --entrypoint /bin/bash rancher/klipper-helm:v0.9.6-build20250616
fc1988754aed:~$ whoami
klipper-helm

fc1988754aed:~$ getent passwd klipper-helm
klipper-helm:x:1000:1000::/home/klipper-helm:/bin/bash

fc1988754aed:~$ ls -lad ~
drwxr-sr-x    1 klipper-helm klipper-helm      4096 Jun 16 14:36 /home/klipper-helm

We did recently bump helm from v3.16.4 to v3.18.1, its possible this is a regression?

[theodiem]

Are you for some reason using a custom deployment of helm-controller, or have you in some way modified the helm job spec or job image?

No. In this case here is the yaml for the helm deployment:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: kubernetes-replicator
  namespace: kube-system
spec:
  jobImage: rancher/klipper-helm:v0.9.6-build20250616
  targetNamespace: system-replicator
  chart: kubernetes-replicator
  repo: https://myhelchartURI
  version: 2.11.1

Its possible something changed in alpine 1.22 but that would be a surprising regression.

Not sure if its alpine, but launching it inside my K3S cluster, HOME is not set as expected:

#kubectl run -it --rm helmdebug --image rancher/klipper-helm:v0.9.6-build20250616 --command -- /bin/bash
If you don't see a command prompt, try pressing enter.

helmdebug:/home/klipper-helm$ whoami
klipper-helm

helmdebug:/home/klipper-helm$ getent passwd klipper-helm
klipper-helm:x:1000:1000::/home/klipper-helm:/bin/bash

helmdebug:/home/klipper-helm$ ls -lad ~
drwxr-xr-x    1 root     root            12 Jun 16 19:28 /

helmdebug:/home/klipper-helm$

You can see that WORKDIR is correct (since the container started on /home/klipper-helm) but you can notice by the PS1 that HOME was not set. Confirmed by the ~ expansion.

If you know the tag of an older image, I'm happy to fire it up (its a ephemeral cluster anyways) to test.

[theodiem]

Not sure if its relevant, but my K3s uses containerd

[brandond]

$HOME has never been explicitly set, as far as I know.

brandond@dev01:~$ docker run --rm -it --entrypoint /bin/bash rancher/klipper-helm:v0.9.5-build20250306 -c 'echo $HOME'
/home/klipper-helm

brandond@dev01:~$ docker run --rm -it --entrypoint /bin/bash rancher/klipper-helm:v0.9.6-build20250616 -c 'echo $HOME'
/home/klipper-helm

[brandond]

I will also note that you shouldn't run a standalone deployment of helm controller with K3s, unless you are for some reason disabling the helm controller that is built in to k3s. Why are you trying to run it standalone?

edit: I see you're just trying to override the job image. If you're going to do that, you should ensure that the job image you're specifying is compatible with the helm-controller release you're using. We haven't updated klipper-helm here yet so what you're doing has not really been tested:

helm-controller/pkg/controllers/chart/chart.go

Line 61 in 5af34a1

DefaultJobImage = "rancher/klipper-helm:v0.9.5-build20250306"

[brandond]

I also can't reproduce this in k3s:

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.5-build20250306 --command -- /bin/bash -c 'echo $HOME'
/home/klipper-helm
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.6-build20250616 --command -- /bin/bash -c 'echo $HOME'
/home/klipper-helm
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl get node -o wide
NAME                         STATUS   ROLES                  AGE    VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-001.example.com   Ready    control-plane,master   4m9s   v1.33.1+k3s1   172.17.0.4    <none>        K3s v1.33.1+k3s1   6.8.0-1027-aws   containerd://2.0.5-k3s1

Can you show the output of kubectl describe node on your host?

[theodiem]

I will also note that you shouldn't run a standalone deployment of helm controller with K3s, unless you are for some reason disabling the helm controller that is built in to k3s. Why are you trying to run it standalone?

I'm not :) ... I have a failed attempt at a helm installation using the manifests addon folder. The helm failed with the messages I've opened the bug from the pod output generated by the "stock" k3s after a k3s server --cluster-init.

Can you show the output of kubectl describe node on your host?
Sure

Name:               cdco-nas-01
Roles:              control-plane,etcd,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    distro=yocto
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=cdco-nas-01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/etcd=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 172.20.0.83
                    etcd.k3s.cattle.io/local-snapshots-timestamp: 2025-06-16T20:02:20Z
                    etcd.k3s.cattle.io/node-address: 172.20.0.83
                    etcd.k3s.cattle.io/node-name: cdco-nas-01-335078c6
                    flannel.alpha.coreos.com/backend-data: null
                    flannel.alpha.coreos.com/backend-type: host-gw
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 172.20.0.83
                    k3s.io/external-ip: 10.192.0.136
                    k3s.io/hostname: cdco-nas-01
                    k3s.io/internal-ip: 172.20.0.83
                    k3s.io/node-args:
                      ["server","--tls-san","10.192.2.166","--tls-san","k3s.localenv.lan","--cluster-domain","k3s.localenv.lan","--flannel-backend","host-gw","-...
                    k3s.io/node-config-hash: KHGN4FUPAUCIXNGPYYPOTTF4R35WSZBZBI2Z2JZYPIRVPTCLSSAQ====
                    k3s.io/node-env: {}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Mon, 16 Jun 2025 17:38:55 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  cdco-nas-01
  AcquireTime:     <unset>
  RenewTime:       Mon, 16 Jun 2025 20:03:10 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  EtcdIsVoter      True    Mon, 16 Jun 2025 19:59:07 +0000   Mon, 16 Jun 2025 17:39:03 +0000   MemberNotLearner             Node is a voting member of the etcd cluster
  MemoryPressure   False   Mon, 16 Jun 2025 20:02:19 +0000   Mon, 16 Jun 2025 17:38:55 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Mon, 16 Jun 2025 20:02:19 +0000   Mon, 16 Jun 2025 17:38:55 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Mon, 16 Jun 2025 20:02:19 +0000   Mon, 16 Jun 2025 17:38:55 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Mon, 16 Jun 2025 20:02:19 +0000   Mon, 16 Jun 2025 17:38:55 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.20.0.83
  ExternalIP:  10.192.0.136
  Hostname:    cdco-nas-01
Capacity:
  cpu:                4
  ephemeral-storage:  8184436Ki
  hugepages-2Mi:      0
  memory:             16368876Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  7961819335
  hugepages-2Mi:      0
  memory:             16368876Ki
  pods:               110
System Info:
  Machine ID:                 92e54f07ef734b5b8199d2eb334263c8
  System UUID:                438f4d56-2da1-0810-8ce7-91b3820288fd
  Boot ID:                    99b5d202-1f83-4014-a818-3bd94284d8a3
  Kernel Version:             6.12.31-yocto-standard
  OS Image:                   MyImage image 5.2+snapshot-2922e9cebf7d3d943e72a0936ccdb0c31a9359b8
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.3
  Kubelet Version:            v1.32.0-k3s1
  Kube-Proxy Version:         v1.32.0-k3s1
PodCIDR:                      172.31.128.0/25
PodCIDRs:                     172.31.128.0/25
ProviderID:                   k3s://cdco-nas-01
Non-terminated Pods:          (3 in total)
  Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
  default                     hdebug                                      0 (0%)        0 (0%)      0 (0%)           0 (0%)         10s
  kube-system                 helm-install-kubernetes-replicator-2dr77    0 (0%)        0 (0%)      0 (0%)           0 (0%)         31s
  kube-system                 metrics-server-8584b5786c-xl7gg             100m (2%)     0 (0%)      70Mi (0%)        0 (0%)         144m
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             70Mi (0%)  0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type    Reason                          Age   From             Message
  ----    ------                          ----  ----             -------
  Normal  Starting                        54s   kube-proxy
  Normal  Starting                        53s   kubelet          Starting kubelet.
  Normal  NodeAllocatableEnforced         52s   kubelet          Updated Node Allocatable limit across pods
  Normal  NodeHasSufficientMemory         52s   kubelet          Node cdco-nas-01 status is now: NodeHasSufficientMemory
  Normal  NodeHasNoDiskPressure           52s   kubelet          Node cdco-nas-01 status is now: NodeHasNoDiskPressure
  Normal  NodeHasSufficientPID            52s   kubelet          Node cdco-nas-01 status is now: NodeHasSufficientPID
  Normal  NodePasswordValidationComplete  51s   k3s-supervisor   Deferred node password secret validation complete
  Normal  RegisteredNode                  32s   node-controller  Node cdco-nas-01 event: Registered Node cdco-nas-01 in Controller

[theodiem]

What I found curious:

# kubectl run -it --rm adebug --image alpine --command -- /bin/sh

If you don't see a command prompt, try pressing enter.

/ # echo $HOME
/root

Trying just for the sake of narrowing it down:

kubectl run -it --rm --restart=Never adebug --image "alpine:3.22" --command -- /bin/sh -c 'echo $HOME'
/root
pod "adebug" deleted

# kubectl run -it --rm --restart=Never adebug --image "rancher/klipper-helm:v0.9.5-build20250306" --command -- /bin/sh -c 'echo $HOME'
/
pod "adebug" deleted

[brandond]

Kubelet Version: v1.32.0-k3s1

You're on a January release of K3s. Can you try updating to the latest 1.32 patch release and see if you still see this behavior?

[theodiem]

Kubelet Version: v1.32.0-k3s1

You're on a January release of K3s. Can you try updating to the latest 1.32 patch release and see if you still see this behavior?

I will try to see if its "just" updating the GIT rev on the build system. Should take ~1h building it.
Still I'm unconvinced, since even firing an alpine container seems to have the HOME env set.

[brandond]

fwiw I see the same behavior on 1.32.0 with both the default and most recent images, however all the helm jobs work fine so I don't think this is the issue.

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.5-build20250306 --command -- /bin/bash -c 'echo $HOME'
If you don't see a command prompt, try pressing enter.
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.6-build20250616 --command -- /bin/bash -c 'echo $HOME'
If you don't see a command prompt, try pressing enter.
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl get node -o wide
NAME                         STATUS   ROLES                  AGE     VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-001.example.com   Ready    control-plane,master   2m14s   v1.32.0+k3s1   172.17.0.4    <none>        K3s v1.32.0+k3s1   6.8.0-1027-aws   containerd://1.7.23-k3s2

Different behavior on the latest 1.32 patch release:

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.5-build20250306 --command -- /bin/bash -c 'echo $HOME'
/home/klipper-helm
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl run --rm -it klipper-helm --restart=Never --image rancher/klipper-helm:v0.9.6-build20250616 --command -- /bin/bash -c 'echo $HOME'
/home/klipper-helm
pod "klipper-helm" deleted

brandond@dev01:~$ kubectl get node -o wide
NAME                         STATUS   ROLES                  AGE    VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION   CONTAINER-RUNTIME
k3s-server-001.example.com   Ready    control-plane,master   105s   v1.32.5+k3s1   172.17.0.4    <none>        K3s v1.32.5+k3s1   6.8.0-1027-aws   containerd://2.0.5-k3s1.32

[theodiem]

Thanks for testing that out before I started looking into the build. So I will try to get the build and if its fixed, I will drop a message in the openembedded lists to see if the version gets bumped.

[theodiem]

So, I did an update to 1.32.5 and the issue persists:

# kubectl describe node
Name:               cdco-nas-01
Roles:              control-plane,etcd,master
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=k3s
                    beta.kubernetes.io/os=linux
                    distro=yocto
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=cdco-nas-01
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/control-plane=true
                    node-role.kubernetes.io/etcd=true
                    node-role.kubernetes.io/master=true
                    node.kubernetes.io/instance-type=k3s
Annotations:        alpha.kubernetes.io/provided-node-ip: 172.20.0.83
                    etcd.k3s.cattle.io/local-snapshots-timestamp: 2025-06-18T00:22:34Z
                    etcd.k3s.cattle.io/node-address: 172.20.0.83
                    etcd.k3s.cattle.io/node-name: cdco-nas-01-15bf31f3
                    flannel.alpha.coreos.com/backend-data: null
                    flannel.alpha.coreos.com/backend-type: host-gw
                    flannel.alpha.coreos.com/kube-subnet-manager: true
                    flannel.alpha.coreos.com/public-ip: 172.20.0.83
                    k3s.io/external-ip: 10.192.0.136
                    k3s.io/hostname: cdco-nas-01
                    k3s.io/internal-ip: 172.20.0.83
                    k3s.io/node-args:
                      ["server","--tls-san","10.192.2.166","--tls-san","k3s.localenv.lan","--cluster-domain","k3s.localenv.lan","--flannel-backend","host-gw","-...
                    k3s.io/node-config-hash: VJRVQOFW4465BSCVSRNO6S5CWUBOUPDM6NXACPVGQ7GFSAR364LQ====
                    k3s.io/node-env: {}
                    node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 18 Jun 2025 00:22:33 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  cdco-nas-01
  AcquireTime:     <unset>
  RenewTime:       Wed, 18 Jun 2025 00:23:45 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  EtcdIsVoter      True    Wed, 18 Jun 2025 00:22:38 +0000   Wed, 18 Jun 2025 00:22:38 +0000   MemberNotLearner             Node is a voting member of the etcd cluster
  MemoryPressure   False   Wed, 18 Jun 2025 00:22:44 +0000   Wed, 18 Jun 2025 00:22:33 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 18 Jun 2025 00:22:44 +0000   Wed, 18 Jun 2025 00:22:33 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 18 Jun 2025 00:22:44 +0000   Wed, 18 Jun 2025 00:22:33 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 18 Jun 2025 00:22:44 +0000   Wed, 18 Jun 2025 00:22:34 +0000   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:  172.20.0.83
  ExternalIP:  10.192.0.136
  Hostname:    cdco-nas-01
Capacity:
  cpu:                4
  ephemeral-storage:  8184436Ki
  hugepages-2Mi:      0
  memory:             16368876Ki
  pods:               110
Allocatable:
  cpu:                4
  ephemeral-storage:  7961819335
  hugepages-2Mi:      0
  memory:             16368876Ki
  pods:               110
System Info:
  Machine ID:                 92e54f07ef734b5b8199d2eb334263c8
  System UUID:                438f4d56-2da1-0810-8ce7-91b3820288fd
  Boot ID:                    99b5d202-1f83-4014-a818-3bd94284d8a3
  Kernel Version:             6.12.31-yocto-standard
  OS Image:                   MyDistro image 5.2+snapshot-2922e9cebf7d3d943e72a0936ccdb0c31a9359b8
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://2.0.3
  Kubelet Version:            v1.32.5-k3s1
  Kube-Proxy Version:         v1.32.5-k3s1
PodCIDR:                      172.31.128.0/25
PodCIDRs:                     172.31.128.0/25
ProviderID:                   k3s://cdco-nas-01
Non-terminated Pods:          (1 in total)
  Namespace                   Name                               CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                               ------------  ----------  ---------------  -------------  ---
  kube-system                 metrics-server-6f4c6675d5-ggbfx    100m (2%)     0 (0%)      70Mi (0%)        0 (0%)         67s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests   Limits
  --------           --------   ------
  cpu                100m (2%)  0 (0%)
  memory             70Mi (0%)  0 (0%)
  ephemeral-storage  0 (0%)     0 (0%)
  hugepages-2Mi      0 (0%)     0 (0%)
Events:
  Type     Reason                          Age                From                   Message
  ----     ------                          ----               ----                   -------
  Normal   Starting                        71s                kube-proxy
  Normal   Starting                        73s                kubelet                Starting kubelet.
  Warning  InvalidDiskCapacity             73s                kubelet                invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory         73s (x2 over 73s)  kubelet                Node cdco-nas-01 status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure           73s (x2 over 73s)  kubelet                Node cdco-nas-01 status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID            73s (x2 over 73s)  kubelet                Node cdco-nas-01 status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced         73s                kubelet                Updated Node Allocatable limit across pods
  Normal   NodeReady                       72s                kubelet                Node cdco-nas-01 status is now: NodeReady
  Normal   NodePasswordValidationComplete  69s                k3s-supervisor         Deferred node password secret validation complete
  Normal   Synced                          68s                cloud-node-controller  Node synced successfully
  Normal   RegisteredNode                  67s                node-controller        Node cdco-nas-01 event: Registered Node cdco-nas-01 in Controller

Then:

# kubectl run -it --rm --restart=Never adebug --image "rancher/klipper-helm:v0.9.6-build20250616" --command -- /bin/sh -c 'echo $HOME'
/
pod "adebug" deleted
# kubectl run -it --rm --restart=Never adebug --image "rancher/klipper-helm:v0.9.5-build20250306" --command -- /bin/sh -c 'echo $HOME'
/
pod "adebug" deleted

But weird enough:

# kubectl run -it --rm --restart=Never adebug --image "alpine:3.22" --command -- /bin/sh -c 'echo $HOME'
/root
pod "adebug" deleted

And here is my config.yaml before running a --cluster-init:

# cat /etc/rancher/k3s/config.yaml
tls-san:
  - 10.192.2.166
  - k3s.localenv.lan
cluster-domain: k3s.localenv.lan

flannel-backend: host-gw
disable:
  - traefik
  - local-storage
  - servicelb
  - coredns

service-cidr: 172.31.0.0/20
cluster-cidr: 172.31.128.0/18
kube-controller-manager-arg:
  - "node-cidr-mask-size=25"
  - "large-cluster-size-threshold=3"

node-label:
  - "distro=yocto"

node-ip: 172.20.0.83
node-external-ip: 10.192.0.136
flannel-iface: netk3s

data-dir: /srv/data2/k3s
kube-proxy-arg:
  - proxy-mode=ipvs
  - ipvs-strict-arp

container-runtime-endpoint: /run/containerd/containerd.sock

Any idea on what to look for would be appreciated