Skip to main content
Version: master

Cgroup Version

Pantavisor runs on either cgroup layout:

  • v1: the hybrid layout Pantavisor mounts today on a device: v1 controllers under /sys/fs/cgroup/<controller> plus cgroup2 at /sys/fs/cgroup/unified. v1 is deprecated.
  • v2: a single cgroup2 hierarchy at /sys/fs/cgroup.

A container says which layouts it supports by carrying explicit lines for each one in its lxc.container.conf. Before LXC reads that file, Pantavisor removes the lines for the layout it is not running on. Pantavisor never rewrites one kind of line into the other.

Choosing the version​

PV_SYSTEM_CGROUP_VERSION takes 1, 2, auto or default:

ValueEmbedded and standaloneAppengine
1Mount the hybrid layout. Fail if that is impossible, e.g. with an LXC that has no v1 support (LXC 7).Fail unless the host runs a hybrid or legacy layout. Also fail with LXC 7.
2Mount cgroup2 at /sys/fs/cgroup. Fail if the kernel has no cgroup2 or no cgroup BPF device support (CONFIG_CGROUP_BPF).Fail unless the host runs a pure cgroup2 layout.
autov2 if the kernel supports it, otherwise v1. Fail if neither works, e.g. LXC 7 on a kernel without v2.Use the layout the host runs.
defaultThe default of the init mode: same as 1.The default of the init mode: same as auto.

Pantavisor checks the kernel for v2 by looking for cgroup2 in /proc/filesystems and by loading a trivial cgroup device BPF program. An invalid value is logged and the default is used.

When the key is not set, or set to default, the default is 1 in embedded and standalone mode and auto in appengine mode, which is the behaviour of Pantavisor before this key existed.

Whenever Pantavisor ends up on v1, it logs a deprecation warning.

The key has no dotted (system.cgroup.version) form.

To choose v2 on a device, set it in pantavisor.config:

PV_SYSTEM_CGROUP_VERSION=2

To see which layout a running system uses:

stat -fc %T /sys/fs/cgroup # cgroup2fs = v2, tmpfs = v1 (hybrid or legacy)

Checking which version is in use​

Pantavisor publishes what auto or default resolved to in two device metadata keys:

KeyValue
pantavisor.cgroup.layouthybrid, legacy or unified (unknown if not detected)
pantavisor.cgroup.version2 for unified, 1 for hybrid and legacy

On the device:

pvcontrol devmeta ls

The same keys show in the device metadata on Pantacor Hub. Pantavisor also logs them at startup in pantavisor.log:

cgroup layout hybrid, using cgroup version 1 (PV_SYSTEM_CGROUP_VERSION=auto)

Cgroup v2 layout​

On v2 the layout is the same on devices and in appengine. In the paths below, <base> is the base cgroup: the cgroup Pantavisor starts in (the root / on a device, the container's own cgroup in appengine).

CgroupHolds
<base>/init.scopePantavisor and every other process that was in the base cgroup, including PID 1 in appengine
<base>/lxc/<name>the container <name>
<base>/lxc.monitor.<name>the LXC monitor of the container

At startup Pantavisor creates <base>/init.scope, moves all processes of the base cgroup into it and only then enables every controller of the base cgroup in <base>/cgroup.subtree_control, so LXC can pass them on to the containers (limits such as pids.max need this). The name init.scope is what makes LXC place containers next to it, in <base>/lxc.

docker exec processes of an appengine container join the cgroup of PID 1, so they run in <base>/init.scope too. To see where everything runs:

for p in /proc/[0-9]*; do echo "${p#/proc/} $(cat $p/cgroup 2>/dev/null) $(cat $p/comm 2>/dev/null)"; done

This is managed automatically; no action needed. v1 and hybrid layouts are unchanged.

Container lines​

Two kinds of line are affected:

KindKeys
v1lxc.cgroup.<controller>.*, e.g. lxc.cgroup.devices.allow, lxc.cgroup.memory.limit_in_bytes
v2lxc.cgroup2.*, e.g. lxc.cgroup2.devices.allow, lxc.cgroup2.memory.max

lxc.cgroup.dir* and lxc.cgroup.relative do not depend on the layout. Pantavisor keeps them in every case and they are not counted below.

A container that works on both layouts carries both kinds:

lxc.cgroup.devices.deny = a
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup2.devices.deny = a
lxc.cgroup2.devices.allow = c 1:3 rwm

Start rule​

Pantavisor removes the lines of the other kind, gives LXC the result, and decides:

Lines removedLines left for the running layoutResult
noneany or nonestart
somesomestart
somenonerefuse to start, and log which lines were removed

The refusal stops a container that was written with a sandbox for one layout from running without any sandbox on the other.

The original lxc.container.conf in the revision is not changed; LXC reads a copy written to <PV_SYSTEM_LXCDIR>/<container>.conf (/run/pantavisor/lxc/<container>.conf by default). The directory is root only and is not mounted into any container. Compare it with the original on a device:

diff /storage/trails/current/<container>/lxc.container.conf /run/pantavisor/lxc/<container>.conf

Each removed line is logged by the pv_lxc module of the Pantavisor log, and so is a refusal:

grep pv_lxc /storage/logs/<revision>/pantavisor/pantavisor.log

devices.allow = a​

A removed lxc.cgroup.devices.allow = a or lxc.cgroup2.devices.allow = a does not count as a removed line: with no device line at all, a container already gets every device its parent cgroup allows, on both layouts and with every LXC version, so removing it takes no sandbox away. Pantavisor logs a warning for each such line, because the line is not needed.

This keeps revisions built with older pvr templates starting on v2: those templates write lxc.cgroup.devices.allow = a into every container that sets no device list.

Pantavisor no longer adds lxc.cgroup2.devices.allow = a on v2.

pvr template​

This part is done by pvr, not by Pantavisor.

The pvr lxc-docker template writes the lines from the container's src.json args. The v1 args exist today; the v2 args are new:

KindDevice listOther cgroup lines
v1PVR_LXC_CGROUP_DEVICES_WHITEPVR_LXC_CGROUP_V1
v2PVR_LXC_CGROUP2_DEVICES_WHITEPVR_LXC_CGROUP_V2

A container that sets both kinds gets both kinds of line. The template stops writing the default lxc.cgroup.devices.allow = a.

meta-pantavisor defaults​

These values are set by the meta-pantavisor layer, not by Pantavisor itself. Without them the embedded default is 1.

BuildPV_SYSTEM_CGROUP_VERSION
Device images with lxc-stable (LXC 6)auto
Device images with lxc-next (LXC 7)2
Device images with lxc-deprecated (LXC 3)1
Appengineauto, set in the Pantavisor appengine config

meta-pantavisor adds no kernel options for this. The BSP decides: with auto, a board whose kernel has cgroup BPF (CONFIG_BPF_SYSCALL, CONFIG_CGROUP_BPF) runs v2, and one without it runs v1 and logs the deprecation warning. When the value is 2, the kernel build fails with an error if those two options are missing.

pantavisor.config ships with the BSP, so a device can change layout in the same revision that brings the new Pantavisor. If a container in that revision cannot start on the new layout, the update fails and the device rolls back to the previous revision.

Open points​

  • Lines that come in through lxc.include are not filtered.
  • Standalone mode does not mount the cgroup tree in Pantavisor today, so the key has no effect there.
  • Containers on v2 see a pure cgroup2 tree under /sys/fs/cgroup. Anything inside a container that reads v1 paths there needs v2 support.