Version: 028-rc11

Automated Test Workflow

Structured testing using test.docker.sh — the test runner bundled inside the pantavisor-appengine-distro build target. Use this for collecting valgrind results and CI validation. For the manual development workflow (quick iteration while coding), including test plans, see development-workflow.md.

Build

Build the distro tarball as described in how-to-build/get-started.md — build target pantavisor-appengine-distro.

When changes are made in meta-pantavisor (test scripts, test.json, expected output, container recipes), a rebuild is required to pick them up. Because BitBake may not detect file-level changes inside a recipe's files/ directory, force a clean rebuild when touching test data:

./kas-container shell kas/build-configs/release/docker-x86_64-scarthgap.yaml -c \
    'bitbake -c cleansstate pantavisor-pvtests-local pantavisor-pvtests-remote pantavisor-appengine-distro pantavisor-bsp pantavisor-default-skel \
     && bitbake -c build pantavisor-appengine-distro'

For quicker iteration, you can also edit files directly inside an already-extracted workdir (e.g. local/lifecycle/seq-non-reboot-updates/resources/test or the output file) without rebuilding. Changes made this way are immediate but ephemeral — they must be ported back to the source tree under recipes-pv/pantavisor-pvtests/files/local/ or recipes-pv/pantavisor-pvtests/files/remote/ to become persistent.

Install

Extract the tarball and load the Docker images as described in how-to-install/docker.md. When working directly on the build machine, the deploy directory already contains an unpacked directory — cd into it and run test.docker.sh without extracting anything.

Remote tests require PH_USER and PH_PASS in the environment (or a sourced .env file).

First-time system setup

On a fresh machine, install all required dependencies (Docker, QEMU, kernel modules, apt packages) before running any tests:

./test.docker.sh install-deps

This is interactive and will prompt before making system changes. In CI set CI_MODE=true to skip the prompt. You only need to run this once per machine; after that, install-docker is sufficient when reinstalling from a new tarball.

The runner uses sudo -n (non-interactive) for several commands during test execution, so those must be allowed without a password in sudoers. Add the following with sudo visudo:

<user> ALL=(ALL) NOPASSWD: /sbin/losetup, /sbin/modprobe, /usr/sbin/iw, /bin/chmod

Running tests

# List available tests
./test.docker.sh -v ls

# Run a specific test (with valgrind)
./test.docker.sh -v run local/core/legacy-config-overload -V

# Run all tests in a category
./test.docker.sh -v run local/lifecycle -V

# Run all local or remote tests
./test.docker.sh -v run local -V
./test.docker.sh -v run remote -V

# Run all tests across all groups
./test.docker.sh -v run -V

The workspace is a temporary directory. Location info (workspace path, log paths) is printed at the start of the run and written to run.log. A copy of run.log is also saved to ./run.log in the current directory for CI consumption.

Workspace layout

<workspace>/
  run.log                           <- location info, one result line per test + inline diffs, SUMMARY
  README.md
  <scope>/<category>/<name>/        <- first attempt
    test.log                        <- full bash-traced output + docker output for this test
    diff                            <- diff (expected vs actual), present only when test failed
    valgrind/
      valgrind.log.<pid>            <- present only when run with -V
  <scope>/<category>/<name>.1/      <- retry attempt 1 (same structure)
  <scope>/<category>/<name>.2/      <- retry attempt 2 (same structure)
  storage/                          <- full Pantavisor on-device storage per test (same naming convention)
    <scope>/<category>/<name>/
      trails/ objects/ disks/ dm-crypt-files/ cache/ boot/ config/ logs/

Note: storage/ is kept on disk for local debugging but is not uploaded to CI artifacts. local/ and remote/ (with per-test logs, diffs, and valgrind results) are uploaded.

Interpreting test results

The run prints location info at the start, then one result line per test (with inline diffs for failures), and ends with a SUMMARY listing every test:

Info: workspace=/tmp/pv_appengine.jBZqVz
Info: readme=/tmp/pv_appengine.jBZqVz/README.md
Info: run log=/tmp/pv_appengine.jBZqVz/run.log
Info: test log=/tmp/pv_appengine.jBZqVz/<scope>/<category>/<name>/test.log
Info: valgrind log=/tmp/pv_appengine.jBZqVz/<scope>/<category>/<name>/valgrind/valgrind.log.<pid>
Info: diff=/tmp/pv_appengine.jBZqVz/<scope>/<category>/<name>/diff

Info: 'local/core/legacy-config-overload' PASSED (23 s)
Info: 'local/lifecycle/reboot-nonreboot-rollback' FAILED (110 s)
--- diff: local/lifecycle/reboot-nonreboot-rollback ---
-expected line
+actual line
--- end diff ---
Info: 'local/runtime/remount-policies' SKIPPED
=======================================================
======================= SUMMARY =======================
=======================================================
Info: 'local/core/legacy-config-overload' PASSED (23 s)
Info: 'local/lifecycle/reboot-nonreboot-rollback' FAILED (110 s)
Info: 'local/runtime/remount-policies' SKIPPED
=======================================================

A failure means actual test output diverged from expected. Lines prefixed with - are expected; lines prefixed with + are what the test produced.

For failing tests, the diff is printed in run.log immediately after the FAILED line, and also saved to <scope>/<category>/<name>/diff. Retry attempts get their own directory (<name>.1/, <name>.2/).

test.log

test.log is a single interleaved stream of everything that happened during a test attempt. It mixes output from four sources:

test.docker.sh (set -x traces) The host-side orchestrator running on the CI runner or developer machine. Visible as ++ docker run ..., ++ allocate_slot, etc. Covers container startup, loop device allocation, and concurrent slot management.

pvtest-run (set -x traces) + resources/test output pvtest-run is the inner test runner inside the tester container. It parses test.json, initialises storage, starts Pantavisor via pv-appengine, waits for it to reach READY, then runs resources/test (the actual test script, with set -x injected at the top). The test script's stdout is captured and diffed against the stored output file; the diff is written to storage/<test_id>/diff and copied to <test_id>/diff in the workspace.

pv-appengine (Pantavisor runtime launcher) Runs inside the tester container. Sets up cgroups, loop devices, and storage mounts, then launches the pantavisor binary in a restart loop to simulate device reboots between update steps.

Pantavisor logs (stdout_direct) Pantavisor is started with PV_LOG_SERVER_OUTPUTS=filetree,stdout_direct. The stdout_direct output mode streams Pantavisor's internal log directly to stdout as each event happens, without buffering. These lines carry the [pantavisor] TIMESTAMP LEVEL -- [module]: message format and are interleaved in real time with the shell traces above. The same log content is also written to storage/<scope>/<category>/<name>/logs/ (kept on disk, not in CI artifacts).

Useful greps on a test.log:

# Pantavisor errors and warnings only
grep " ERROR\b\| WARN\b" test.log

# Just the test script execution (resources/test set -x traces)
grep "^+ \|^++ " test.log | tail -50

Valgrind logs

With -V, each process gets its own valgrind.log.<pid> file under <scope>/<category>/<name>/valgrind/ (and <name>.1/valgrind/ for retries). Pantavisor forks heavily via LXC, so there will be many files. The main Pantavisor worker is typically the largest:

ls -S <workspace>/local/lifecycle/reboot-nonreboot-rollback/valgrind/ | head -3
grep -E "definitely lost|possibly lost|ERROR SUMMARY" valgrind.log.<largest-pid>

definitely lost — real leaks, investigate
possibly lost — typically PV buffer pools (pv_buffer_init); consistent across all tests at ~3.7 MB, not a regression
ERROR SUMMARY — mostly Syscall param warnings from liblxc (openat2/mount), not pantavisor code
No summary at the end of a file means the process was killed before valgrind finished flushing

Debugging a failing test

# Interactive shell — Pantavisor starts normally; shell opens once it reaches READY
# (and claims the device if credentials are configured).
# Use when Pantavisor boots fine but you want to inspect the running state.
./test.docker.sh -v run local/core/legacy-config-overload -i

# Manual shell — container starts but Pantavisor does NOT run.
# Use when Pantavisor fails to reach READY and you need to debug the startup sequence.
./test.docker.sh -v run local/core/legacy-config-overload -m

Both -i and -m require a specific leaf test path.

Authoring and updating tests

Adding a new test from scratch

Test data lives in the meta-pantavisor source tree under:

recipes-pv/pantavisor-pvtests/files/local/    # local tests
recipes-pv/pantavisor-pvtests/files/remote/   # remote tests

Each test is a directory at <scope>/<category>/<name>/ containing test.json, resources/test, and an output file.

1. Create the test directory using the add command from the workdir:

# From the workdir (e.g. workdir/appengine-<commit>/):
./test.docker.sh add local/lifecycle/my-new-test
# Info: New test created at: .../local/lifecycle/my-new-test

This copies all templates (test.json, resources/test, resources/ready) and sets permissions. Once you have edited the test, port it back to the source tree:

cp -r <workdir>/local/lifecycle/my-new-test \
      recipes-pv/pantavisor-pvtests/files/local/lifecycle/

2. Edit test.json:

Field	Purpose	Notes
`#spec`	always `"pv-test@1"`	do not change
`description`	human-readable summary	keep it short
`setup.cmdline`	kernel cmdline overrides	`""` if not needed
`setup.env`	space-separated `KEY=VALUE` env vars for Pantavisor	e.g. `"PV_WDT_MODE=disabled PV_SECUREBOOT_MODE=disabled"`
`setup.pantavisor.config`	path to a custom `pantavisor.config`, or `""`	e.g. `"resources/pantavisor.config"`
`setup.pvs`	glob for PVS signing key tarballs	keep `"../../common/pvs/*.tar.gz"` for local; `""` for remote
`setup.containers.control`	name of the container used as control plane	usually `"pvr-sdk"`
`setup.containers.tarballs`	list of container pvrexport tarballs	always include `bsp.tgz` and `pvr-sdk.tgz`; add extra containers as needed
`setup.containers.urls`	OTA container URLs (remote tests)	`[]` for local tests
`setup.ready-script`	script to run once Pantavisor reaches READY	`""` if not needed; `"resources/ready"` otherwise
`setup.self-claim`	remote tests only: auto-claim the device	`"true"`
`test-script`	path to the test script	`"resources/test"`
`skip`	exclude test from runs	`"false"` normally; `"true"` to disable

3. Write resources/test:

#!/bin/sh

source /work/scripts/utils
. /opt/pantavisor/set_env

# Use pventer to run commands inside a container; stdout is diff-ed against `output`
pventer -c pvr-sdk pvcontrol config ls | jq -M -r '.["policy"]'

Guidelines (from GEMINI.md conventions):

Always source utils and set_env at the top — they set up the test environment
Use pventer -c <container> <cmd> for commands inside containers
Use pvcontrol and pvcurl for the pv-ctrl API
Output determinism: pipe JSON through jq -M (compact, sorted) and use tr -d '\r' to strip carriage returns — the runner diffs stdout byte-for-byte
Keep tests independent: each test starts from a clean container and storage state

4. Generate the output file (never edit manually):

./test.docker.sh -v run $SCOPE/$CATEGORY/$NAME -o

This writes output into the extracted workdir at <workdir>/$SCOPE/$CATEGORY/$NAME/output.

5. Copy output back to the source tree:

cp <workdir>/$SCOPE/$CATEGORY/$NAME/output \
   recipes-pv/pantavisor-pvtests/files/$SCOPE/$CATEGORY/$NAME/output

6. Rebuild and verify (see Build above for the full cleansstate command):

./kas-container shell kas/build-configs/release/docker-x86_64-scarthgap.yaml -c \
    'bitbake -c cleansstate pantavisor-pvtests-local pantavisor-pvtests-remote pantavisor-appengine-distro \
     && bitbake -c build pantavisor-appengine-distro'
./test.docker.sh -v run $SCOPE/$CATEGORY/$NAME

Iterate between steps 4–6 until the test passes cleanly.

7. Add the test to the test plan table below and mark [x] in TODO.md.

Updating expected output for an existing test

After a behaviour change makes an existing test fail with a known-good diff, regenerate its output:

./test.docker.sh -v run local/core/legacy-config-overload -o
cp <workdir>/local/core/legacy-config-overload/output \
   recipes-pv/pantavisor-pvtests/files/local/core/legacy-config-overload/output

Then rebuild and verify as above.

Adding a new container for a test

When a test needs a container that does not exist yet in local/common/tarballs/:

1. Create the recipe in recipes-containers/pv-examples/<name>.bb — use pv-example-app.bb as a reference:

SUMMARY = "..."
LICENSE = "MIT"
LIC_FILES_CHKSUM = "file://${COMMON_LICENSE_DIR}/MIT;md5=0835ade698e0bcf8506ecda2f7b4f302"
inherit image container-pvrexport
IMAGE_BASENAME = "<name>"
IMAGE_INSTALL = "busybox"
IMAGE_FEATURES = ""
IMAGE_LINGUAS = ""
NO_RECOMMENDATIONS = "1"
PVRIMAGE_AUTO_MDEV = "0"
SRC_URI += "file://<script>.sh"
install_scripts() {
    install -d ${IMAGE_ROOTFS}${bindir}
    install -m 0755 ${WORKDIR}/<script>.sh ${IMAGE_ROOTFS}${bindir}/<entrypoint>
}
ROOTFS_POSTPROCESS_COMMAND += "install_scripts; "
PVR_APP_ADD_EXTRA_ARGS += "--config=Entrypoint=/usr/bin/<entrypoint>"

2. Register it in recipes-pv/pantavisor/pantavisor-appengine-distro.bb:

Add to do_create_tarball[depends]:

do_create_tarball[depends] += "<name>:do_image_complete"

Add a copy block inside do_create_tarball():

for f in ${DEPLOY_DIR_IMAGE}/<name>.pvrexport.tgz; do
    if [ -e "$f" ]; then
        cp -v "$f" "${STAGING_DIR}/local/common/tarballs/<name>.tgz"
        break
    fi
done

3. Reference in test.json:

"tarballs": [
  "../../common/tarballs/bsp.tgz",
  "../../common/tarballs/pvr-sdk.tgz",
  "../../common/tarballs/<name>.tgz"
]

4. Rebuild as in step 6 above.

test.docker.sh flags reference

Global options (before the command):

Flag	Description
`-v`, `--verbose`	Enable debug output and print a results summary at the end
`-d <dir>`, `--dir <dir>`	Use `<dir>` as the pvtest source directory (overrides `PVTEST_DIR` env)

run arguments (after the path selector):

Flag	Description
`-V`, `--valgrind`	Run Pantavisor under valgrind; results saved to `<tmpdir>/valgrind/`
`-i`, `--interactive`	Open a shell once Pantavisor reaches READY (device claimed if configured). Use to inspect a working system. Requires a specific leaf test path.
`-m`, `--manual`	Open a shell without starting Pantavisor. Use when PV fails to reach READY and you need to debug startup. Requires a specific leaf test path.
`-o`, `--overwrite`	Create or overwrite the expected test output (use when authoring or updating tests)
`-n`, `--netsim`	Enable wireless network simulation via `mac80211_hwsim` (experimental)

Exit codes: 0 = PASSED, 1 = FAILED, 2 = ABORTED

Test plan

Tests are organized by scope (local / remote) and category. The table below tracks implementation status.

local — tests running entirely within the appengine container

core

Test	Description	Done
`local/core/legacy-config-overload`	Legacy configuration overload	✓
`local/core/modern-config-overload`	Modern configuration overload (Env/Cmdline)	✓
`local/core/invalid-config-values`	Invalid Configuration Values Handling
`local/core/rootfs-namespace`	Rootfs namespace (mounts, symlinks, etc.)

lifecycle

Test	Description	Done
`local/lifecycle/reboot-nonreboot-rollback`	Reboot, non-reboot and rollback updates	✓
`local/lifecycle/seq-non-reboot-updates`	Sequential non-reboot updates	✓
`local/lifecycle/power-loss-during-update`	Power Loss During Update
`local/lifecycle/shared-object-restart-policies`	Shared object update with distinct restart policies
`local/lifecycle/auto-recovery-restart`	Auto-recovery restart on failure
`local/lifecycle/auto-recovery-retries-rollback`	Auto-recovery retries exhaustion during TESTING triggers rollback
`local/lifecycle/auto-recovery-stable-timeout`	Auto-recovery stable timeout holds commit
`local/lifecycle/auto-recovery-never-stops`	Auto-recovery policy never stops container after retries
`local/lifecycle/auto-recovery-stabilize`	Stabilize pattern: container fails N times then becomes stable
`local/lifecycle/auto-recovery-always-restart`	Always-restart policy on any exit code
`local/lifecycle/auto-recovery-group-inheritance`	Group-level auto-recovery policy inherited by containers
`local/lifecycle/auto-recovery-container-override`	Container auto-recovery overrides group (all-or-nothing)
`local/lifecycle/auto-recovery-backoff-duration`	Backoff duration resets retry cycle after exhaustion

runtime

Test	Description	Done
`local/runtime/invalid-state-json`	Invalid State JSON
`local/runtime/large-state-json`	Large State JSON (100+ containers)
`local/runtime/container-groups-startup`	Container Groups and Startup Order
`local/runtime/container-storage-persistence`	Container Storage Persistence
`local/runtime/config-overlay`	Configuration Overlay
`local/runtime/resource-constraints`	Resource Constraints (CPU/Mem)
`local/runtime/status-goal-success-failure`	Status Goal Success and Failure	✓
`local/runtime/container-exports`	Container Exports to Host	✓
`local/runtime/remount-policies`	Remount Policies (PV_REMOUNT_POLICY)	✓

control

Test	Description	Done
`local/control/basic-endpoints`	Basic Endpoints (Containers, Objects, etc.)	✓
`local/control/invalid-signal-handling`	Invalid Signal Handling
`local/control/local-run-command`	Local Run Command
`local/control/ssh-override`	SSH Override
`local/control/object-step-management`	Object & Step Management
`local/control/metadata-manipulation`	Metadata Manipulation
`local/control/pvcontrol-pvcurl`	pvcontrol & pvcurl tool verification

xconnect

Test	Description	Done
`local/xconnect/unix-sockets`	Unix Sockets (UDS proxying)
`local/xconnect/rest-over-uds`	REST-over-UDS (Identity headers)
`local/xconnect/dbus`	D-Bus (Policy mediation)
`local/xconnect/drm`	DRM (Graphics node injection)
`local/xconnect/wayland`	Wayland (Isolated UI rendering)

security

Test	Description	Done
`local/security/strict-secure-boot`	Strict Secure Boot (Unsigned rejection)	✓
`local/security/container-roles`	Container Roles (mgmt vs nobody access)	✓
`local/security/oem-secureboot`	OEM Secureboot (OEM key validation)	✓
`local/security/object-checksum`	Object Checksum Validation	✓
`local/security/lenient-secure-boot`	Lenient Secure Boot
`local/security/encrypted-storage`	Encrypted Storage (LUKS/dm-crypt)
`local/security/secureboot-sig-0x30`	Secure Boot when signature starts with 0x30

services

Test	Description	Done
`local/services/log-output-formats`	Log Output Formats (filetree/singlefile)
`local/services/on-demand-gc`	On-Demand Garbage Collection	✓
`local/services/tsh-daemon`	tsh daemon management & log capture
`local/services/log-rotation`	Log rotation functionality
`local/services/ipam-single-pool`	Single IPAM pool — container gets IP from pool
`local/services/ipam-multi-pool`	Two IPAM pools — correct address assignment
`local/services/ipam-collision`	Conflicting pool addresses detected and rejected
`local/services/ipam-invalid`	Invalid IPAM config rejected gracefully
`local/services/ipam-lxcbr`	IPAM with lxcbr bridge networking

remote — tests requiring Pantahub connectivity

core

Test	Description	Done
`remote/core/encrypted-pantahub-config`	Encrypted `pantahub.config` handling	✓
`remote/core/unencrypted-pantahub-config`	Unencrypted `pantahub.config` handling	✓

lifecycle

Test	Description	Done
`remote/lifecycle/simultaneous-updates`	Successful Multiple Simultaneous Remote Updates	✓
`remote/lifecycle/insufficient-disk-space`	Update with Insufficient Disk Space	✓
`remote/lifecycle/rollback-cloud-status`	Trigger rollback and verify cloud status	✓
`remote/lifecycle/update-retries-pv-crash`	Update retries when PV crashes	✓
`remote/lifecycle/update-retries-gc-pressure`	Update retries when PV crashes with GC pressure	✓
`remote/lifecycle/claim-after-local-updates`	Claim after local updates with random artifacts

control

Test	Description	Done
`remote/control/manual-claim`	Manual Device Claim	✓
`remote/control/auto-claim`	Automatic Device Claim	✓
`remote/control/always-remote-disabled`	Always Remote Disabled
`remote/control/always-remote-enabled`	Always Remote Enabled	✓
`remote/control/device-user-metadata`	Device/User Metadata Exchange

services

Test	Description	Done
`remote/services/ph-logger-cloud-push`	`ph-logger` cloud push	✓

Build​

Install​

First-time system setup​

Running tests​

Workspace layout​

Interpreting test results​

test.log​

Valgrind logs​

Debugging a failing test​

Authoring and updating tests​

Adding a new test from scratch​

Updating expected output for an existing test​

Adding a new container for a test​

test.docker.sh flags reference​

Test plan​

local — tests running entirely within the appengine container​

core​

lifecycle​

runtime​

control​

xconnect​

security​

services​

remote — tests requiring Pantahub connectivity​

core​

lifecycle​

control​

services​

Build

Install

First-time system setup

Running tests

Workspace layout

Interpreting test results

test.log

Valgrind logs

Debugging a failing test

Authoring and updating tests

Adding a new test from scratch

Updating expected output for an existing test

Adding a new container for a test

test.docker.sh flags reference

Test plan

local — tests running entirely within the appengine container

core

lifecycle

runtime

control

xconnect

security

services

remote — tests requiring Pantahub connectivity

core

lifecycle

control

services