api, cgroupv2: skip setting the devices cgroup

[giuseppe]

The Kubelet uses libcontainer/cgroups to setup cgroups. It would be nice to have a way to skip setting the devices cgroup at all.

[mrunalp]

@kolyshkin @AkihiroSuda fyi

[cyphar]

😬 Is there a reason for wanting to skip the devices cgroup? The hard requirement for the devices cgroup is to make sure we have a fail-secure setup (if you don't set the devices whitelist, you will allow users to do all sorts of scary things). We could make it so that this requirement is only present for runc but I'm not sure how ugly that would become...

[giuseppe]

the Kubelet uses the libcontainer code to create the parent cgroups (e.g. /sys/fs/cgroup/kubepods). Each container will have its own cgroup under /sys/fs/cgroup/kubepods where the device cgroup is configured.
I don't think it is a problem on cgroup v1, but on cgroup v2, every time the Kubelet restarts and tries to configure the cgroup, it leaks an eBPF program

[cyphar]

Ah okay. Yeah we might need to add a way to configure that. The only important thing is that this should be strictly opt-out with a fairly large warning sign next to the configuration option.

[kolyshkin]

Addressed this one in #2490

[cyphar]

As discussed in #2490, I believe that the cgroupv2 eBPF issue is a runc bug that we should fix anyway (tracked by #2366). Would the Kubernetes folks still need this if we fixed the cgroupv2 eBPF issue?

[giuseppe]

the devices cgroup is not used at all by Kubernetes: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/container_manager_linux.go#L378-L385

Since the cost of running such eBPF program is close to 0, I am fine if the underlying issue is addressed instead of offering a different API.

[odinuge]

the devices cgroup is not used at all by Kubernetes:

Well, that is kinda true, but only applies to kubelet and its usage of libcontainer.

However, Kubernetes is implicitly using it via the CRI api since most CRI implementations use runc by default. As I understand the cgroup.SkipDevices, is only for users of the go api, not the binary.

When running kubernetes with with eg. containerd as the runtime (via CRI), it uses the runc binary. Kubernetes is also constantly updating the cpus-set for containers via a component called the "CPU Manager". When trying to run that setup with cgroup v2, containerd start replying to container resource updates (CRI update consists of only CPUset, but unsure what command containerd sends to runc, but can take a look at that) with runc did not terminate successfully: failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): argument list too long. This makes it essentially block cpuset updates for containers when using runc.

Can x-post this to #2366 tho, since it might be useful in that context.