Cgroup V2: cAdvisor Exposes Misleading IO metrics (container_fs_*_bytes_total)

[iyashu]

Few IO related metrics are always showing zero values (e.g container_fs_writes_bytes_total, container_fs_reads_bytes_total ) in cgroup v2 enabled node even there is one container writing heavily into some disk in that node. Although, I can see the io.stats file under that cgroup directory is showing the correct values.

while [ 1 ]; do echo $(cat io.stat | grep "254:3"); sleep 1; done
254:3 rbytes=598016 wbytes=227002702848000 rios=146 wios=124342581 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227003599380480 rios=146 wios=124343436 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227004496056320 rios=146 wios=124344326 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227005448163328 rios=146 wios=124345234 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227006373007360 rios=146 wios=124346116 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227007340843008 rios=146 wios=124347039 dbytes=0 dios=0
254:3 rbytes=598016 wbytes=227008280367104 rios=146 wios=124347935 dbytes=0 dios=0

In cgroup v1, I can see that the same metrics are being surfaced with the expected values in prometheus, but thats not the case with cgroup v2.

Original Issue: kubernetes/kubernetes#102285

• Cadvisor Version: v0.37.4 (even the latest version is having the same issue)
• OS (e.g: cat /etc/os-release): Debian GNU/Linux 10 (buster)
• Kernel (e.g. uname -a): 5.10.0-0.bpo.3-cloud-amd64

[dims]

@iyashu how exactly did you switch on cgroup v2? (do you have a list of steps?)

[iyashu]

@dims we enabled cg2 by adding systemd.unified_cgroup_hierarchy=1 under the default grub file.

    sed -i -E "s/^GRUB_CMDLINE_LINUX_DEFAULT=\"\"$/GRUB_CMDLINE_LINUX_DEFAULT=\"systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all \"/g" /etc/default/grub
    update-grub

[iyashu]

Alrighty, I spent some time in debugging the issue by adding few log stmts. In cadvisior codebase, we are populating container_fs_writes_bytes_total metric value with Write key of DiskIO.IoServiceBytes map which gets populated by runc GetStats api.

cadvisor/metrics/prometheus.go

Lines 664 to 675 in ef7e64f

	name: "container_fs_writes_bytes_total",
	help: "Cumulative count of bytes written",
	valueType: prometheus.CounterValue,
	extraLabels: []string{"device"},
	getValues: func(s *info.ContainerStats) metricValues {
	return ioValues(
	s.DiskIo.IoServiceBytes, "Write", asFloat64,
	nil, nil,
	s.Timestamp,
	)
	},
	}, {

But in runc fs2 GetStats api implementation, we are populating the value with key write (small w case) for wbytes value of io.stats file. Does it make sense to fix it at runc codebase to make it consistent with cgroup v1?

https://github.com/opencontainers/runc/blob/e005fee9d37f3ee35cfc6d4cfee687043656928f/libcontainer/cgroups/fs2/io.go#L122-L124

[dims]

@iyashu build cadvisor locally with that change and see if that fixes the problem?

[iyashu]

Yep, it fixes up the problem after i update the runc to use Write instead of write and recompile cadvisor with updated runc dependency (containing the fix).

[dims]

@iyashu please log a issue + PR in the https://github.com/opencontainers/runc repo. Nice job!

[dims]

does read have to be Read as well there?

[dims]

Also point them to the examples here : https://github.com/opencontainers/runc/blob/c061bc78aae9b39b8ac5d3011e55736b20288cb3/libcontainer/cgroups/fs/blkio.go#L72-L101

[iyashu]

Yes, read have to be Read. Another thing is the number of read or write IOs are not provided by runc in case of cg2. In cgroup v1, we used get container_fs_writes_total metric (populated via runc stats.BlkioStats.IoServicedRecursive), but with cg2 it is not populated. I'll try to fix that as well in runc repo. Thanks @dims for acknowledging and helping out promptly.