Complicated cgroup manager instantiation (and how to fix it)

[kolyshkin]

Currently, libcontainer has 4 different cgroup managers:

• cgroup v1 cgroupfs-based (aka fs)
• cgroup v2 cgroupfs-based (aka fs2)
• cgroup v1 systemd-based (aka systemd v1)
• cgroup v2 systemd-based (aka systemd v2)

In addition, 3 of the above also implement optional "rootless" functionality,
which can be enabled during manager's instantiation.

Because of the above, there is no easy way to "get a cgroup manager",
and the code that instantiates one is usually complicated. For example,
these almost 100 lines of code:

runc/libcontainer/factory_linux.go

Lines 58 to 150 in f67f1ef

	func getUnifiedPath(paths map[string]string) string {
	path := ""
	for k, v := range paths {
	if path == "" {
	path = v
	} else if v != path {
	panic(fmt.Errorf("expected %q path to be unified path %q, got %q", k, path, v))
	}
	}
	// can be empty
	if path != "" {
	if filepath.Clean(path) != path || !filepath.IsAbs(path) {
	panic(fmt.Errorf("invalid dir path %q", path))
	}
	}
	return path
	}
	func systemdCgroupV2(l *LinuxFactory, rootless bool) error {
	l.NewCgroupsManager = func(config *configs.Cgroup, paths map[string]string) cgroups.Manager {
	return systemd.NewUnifiedManager(config, getUnifiedPath(paths), rootless)
	}
	return nil
	}
	// SystemdCgroups is an options func to configure a LinuxFactory to return
	// containers that use systemd to create and manage cgroups.
	func SystemdCgroups(l *LinuxFactory) error {
	if !systemd.IsRunningSystemd() {
	return errNoSystemd
	}
	if cgroups.IsCgroup2UnifiedMode() {
	return systemdCgroupV2(l, false)
	}
	l.NewCgroupsManager = func(config *configs.Cgroup, paths map[string]string) cgroups.Manager {
	return systemd.NewLegacyManager(config, paths)
	}
	return nil
	}
	// RootlessSystemdCgroups is rootless version of SystemdCgroups.
	func RootlessSystemdCgroups(l *LinuxFactory) error {
	if !systemd.IsRunningSystemd() {
	return errNoSystemd
	}
	if !cgroups.IsCgroup2UnifiedMode() {
	return errors.New("cgroup v2 not enabled on this host, can't use systemd (rootless) as cgroups manager")
	}
	return systemdCgroupV2(l, true)
	}
	func cgroupfs2(l *LinuxFactory, rootless bool) error {
	l.NewCgroupsManager = func(config *configs.Cgroup, paths map[string]string) cgroups.Manager {
	m, err := fs2.NewManager(config, getUnifiedPath(paths), rootless)
	if err != nil {
	panic(err)
	}
	return m
	}
	return nil
	}
	func cgroupfs(l *LinuxFactory, rootless bool) error {
	if cgroups.IsCgroup2UnifiedMode() {
	return cgroupfs2(l, rootless)
	}
	l.NewCgroupsManager = func(config *configs.Cgroup, paths map[string]string) cgroups.Manager {
	return fs.NewManager(config, paths, rootless)
	}
	return nil
	}
	// Cgroupfs is an options func to configure a LinuxFactory to return containers
	// that use the native cgroups filesystem implementation to create and manage
	// cgroups.
	func Cgroupfs(l *LinuxFactory) error {
	return cgroupfs(l, false)
	}
	// RootlessCgroupfs is an options func to configure a LinuxFactory to return
	// containers that use the native cgroups filesystem implementation to create
	// and manage cgroups. The difference between RootlessCgroupfs and Cgroupfs is
	// that RootlessCgroupfs can transparently handle permission errors that occur
	// during rootless container (including euid=0 in userns) setup (while still allowing cgroup usage if
	// they've been set up properly).
	func RootlessCgroupfs(l *LinuxFactory) error {
	return cgroupfs(l, true)
	}

Also, every user of libcontainer/cgroups has to reimplement something like the above. This includes kubernetes, cadvisor, and our tests.

I propose to solve this by

1. Moving rootless boolean property from the cgroup manager to cgroup config.
2. Adding Systemd boolean property to cgroup config.
3. Introducing a new package, libcontainer/cgroups/manager, with the following functions:

// New returns the instance of a cgroup manager, which is chosen
// based on the local environment (whether cgroup v1 or v2 is used)
// and the config (whether config.Systemd is set or not).
func New(config *configs.Cgroup) (cgroups.Manager, error) {
       return NewWithPaths(config, nil)
}
 
// NewWithPaths is similar to New, and can be used in case cgroup paths
// are already well known, which can save some resources.
//
// For cgroup v1, the keys are controller/subsystem name, and the values
// are absolute filesystem paths to the appropriate cgroups.
//
// For cgroup v2, the only key allowed is "" (empty string), and the value
// is the unified cgroup path.
func NewWithPaths(config *configs.Cgroup, paths map[string]string) (cgroups.Manager, error)

With this in place, the above code (first example) is replaced with something like

          cm, err := manager.New(config.Cgroups)
          if err != nil {
                  return err
          }

In a similar way, external users (kubernetes, cadvisor) can simplify cgroup manager instantiation.

[kolyshkin]

The fix is implemented as part of PR #3131.