Lifecycle testing — clean install and install → upgrade¶

Every other gate in this repo tests code in isolation. make test-container, the *-roundtrip-container harnesses, make tui-driver-selftest — each builds the current tree and exercises it. None of them ever installs a release, upgrades it, and checks what the machine looks like afterwards.

That is where the bugs users actually hit live. They are not logic bugs; they are lifecycle bugs, and they need two versions and a real machine to exist at all:

	what escaped
#1921	a new client against an old daemon hard-failed (`unknown field "tab_id"`). The daemon is upgraded independently of its clients, so this is only reachable across a version boundary.
#1916	`af reset` SIGKILLed a wedged daemon; the service manager relaunched it mid-wipe. Found by a user.
#796	the daemon demoted from supervised (a systemd unit) to an ad-hoc child: a daemon still runs and still answers, while the unit sits inactive/dead.

A dev box is the opposite of the environment those need: it has an installed unit, a live daemon, and months of state. "Works here" says nothing about a new machine, which has no config, no unit, no home, no daemon.

scripts/lifecycle.sh builds that machine from nothing, then upgrades it.

Running it¶

# The whole gate, container-fenced. The only way to run it on a dev box.
make lifecycle-container

# Narrow it while iterating.
make lifecycle-container LIFECYCLE_SCENARIO=scenario-a
make lifecycle-container LIFECYCLE_SCENARIO=scenario-b

It needs network: it downloads real published release tarballs. That is the point — the whole gate is about the boundary between two real releases — but it also means this is the one testbox target that is not hermetic, which is why it is wired nightly rather than per-PR (.github/workflows/lifecycle.yml). It also runs on a PR that touches the gate's own files, so a change to the harness proves itself.

CI authenticates the harness's release-list request with the scoped workflow GITHUB_TOKEN, so that request does not consume the shared runner IP's unauthenticated quota. Product release discovery intentionally remains anonymous and never reads that ambient token; the harness therefore treats a transport failure, quota response (403/429), or GitHub 5xx from either discovery path as SKIP / could not verify, not as “upgrade broken.” Malformed release data, a 404, and every failure after discovery remain FAIL, so availability handling cannot launder a product or release regression into a green result.

The scenarios¶

A — clean first run¶

The machine a new user has: no config.toml, no AF home, no autostart unit, no daemon. Nothing in af may assume any of them exist. It asserts the preconditions first (so a "clean run" that wasn't clean cannot pass quietly), then: af version works, af doctor reports zero FAIL a new user cannot act on (missing agents are WARNs with a fix line — correct), the TUI renders its first frame on a virgin home without panicking, and the daemon comes up exactly once.

B — install → upgrade¶

Install a real previous release, put sessions on it, upgrade the way a user actually does, assert the machine is coherent. Both upgrade paths are covered, because they are different code paths into the same restart:

af upgrade
the launch-time auto-update (commands/root.go → autoUpdateOnLaunch), which is gated on stdout being a TTY and so can only be driven through a real terminal — a tmux pane, never a plain exec.

#	assertion
1	the daemon is restarted onto the new binary — not left running the old one
2	client version == daemon version, queried rather than inferred
3	exactly one daemon afterwards; no orphan survives
4	if a unit was installed, it is still the supervisor — not demoted to an ad-hoc child, and the unit is not left inactive/dead
5	pre-existing sessions survive (count before == after; none Lost)
6	`af doctor` reports zero FAILs afterwards

The release pair is resolved from the API at run time (the two newest non-prereleases), not pinned: nothing here asserts a specific version, so the gate keeps testing the current boundary as releases cut.

How assertion 2 queries the daemon¶

The daemon answers a control-socket ping with its own version (#1920 added PingResponse.Version), and af daemon status --json surfaces it as .data.version. That is what assertion 2 reads. It reports the responding daemon's version or nothing at all — never the client's — which is the property the assertion needs: after an upgrade the on-disk binary is N while a daemon that was never restarted still answers as N-1, and a version read off the installed binary would report N and call the skew healthy.

When the daemon reports nothing, lc_daemon_version falls back to the image it is actually executing: copy /proc/<pid>/exe and ask it. /proc/<pid>/exe still resolves to those bytes even though the path now reads (deleted). The fallback is feature-detected rather than version-gated, because two different daemons answer with nothing and only one of them is old:

one older than v1.0.206, before af daemon status --json grew the member (PingResponse.Version itself landed in v1.0.200). Scenario B installs the two newest stable releases, so this is only reachable if the boundary is ever pinned backwards — but the harness must not quietly stop measuring if it is;
one that refuses this client's ping — which is the #1921 skew the gate exists to catch. The route that goes quiet is exactly the route under test, so the fallback is load-bearing rather than legacy politeness.

Neither route can read the new binary on disk, so a daemon left on the old bytes cannot hide behind it. If both come back empty, lc_assert_no_version_skew fails the run: lc_assert_eq "" "" would otherwise report "no skew" having compared nothing against nothing, and that green is worse than a red one.

lc_doctor_fail_count feature-detects af doctor --json the same way: the structured summary when the flag is there (it is, since #1920), the text Summary: line for an older N-1 client.

Isolation¶

The harness is destructive by design — it installs binaries, registers autostart units, upgrades af over itself, and stops daemons. It refuses to run unless both are true:

AF_LIFECYCLE_DISPOSABLE=1 is set explicitly, and
the environment positively looks disposable — /.dockerenv (container) or CI=true.

A shared dev box has neither, so it refuses there even with the opt-in set. The workspace is validated too: it can never be, or sit inside, the real AF home. Every daemon it signals is proven to serve its own throwaway home by reading /proc/<pid>/environ — so it cannot touch a real daemon, or another user's.

Proving the gate still bites¶

Most of the bugs above are fixed, so a green run proves little on its own — the value is the ones it catches next. AF_LIFECYCLE_INJECT deliberately breaks the machine so the assertions can be watched failing:

AF_LIFECYCLE_INJECT=skip-daemon-restart make lifecycle-container LIFECYCLE_SCENARIO=scenario-b

injection	reconstructs	must fail
`skip-daemon-restart`	the #1921 machine: binary swapped to N, daemon never restarted — new client, old daemon, everything apparently "running"	assertions 1 and 2
`unhealthy-session`	a session left in a non-healthy state across the upgrade (`Archived(6)/LiveArchived(5)`)	assertion 5

Re-run these whenever you change the harness. An assertion that cannot be watched failing is not evidence — and this gate has caught itself twice on exactly that:

the Lost check tested status/liveness == 4 (Dead/LiveDead), both write-never since #1108 — deaths record Lost, and persisted Dead is rewritten to Lost on load. It could not match anything on any machine, and reported "0 sessions Lost: PASS" forever. It is now fail-closed: it asks "is every session in a state I recognise as healthy?", so an unknown or drifted enum value fails LOUDLY instead of quietly matching nothing;
an unscoped sessions list counted the wrong project's sessions and passed locally by accident (see the --repo note above).

Why unhealthy-session archives rather than killing a pane: it can't be Lost on demand. Killing a session's tmux does not strand it — the daemon restores it within ~4s (measured: Running→Ready, and it heals even with the worktree deleted, #1108's restore loop). So an injection built on that would quietly inject nothing, which is the same vacuity in a different hat. Archive is durable and lands on a real non-healthy state.

Where assertion #4 (supervision) actually runs¶

This is the assertion the gate most needs — an upgrade demoting the daemon off its unit (#796) is invisible to every other check here, because the demoted daemon keeps running and keeps answering. So it gets its own rules.

It runs on the CI native leg, against a real systemd user manager, and it passes there today:

PASS  assertion 4: the unit is still active (= active)
PASS  assertion 4: af still sees the autostart unit (= true)
PASS  assertion 4: the running daemon IS the unit's child (MainPID=7712) — not demoted

That last line is the one that catches the demotion: it compares the daemon that is running against the daemon systemd owns. A demoted daemon still answers pings; it just is not the unit's child any more.

It cannot run in the test container, and that is not a policy choice:

approach	verdict
`systemd --user` standalone in the container	impossible — it refuses without PID 1 systemd: "Trying to run as user instance, but the system has not been booted with systemd."
full systemd as PID 1 in the container	works, but needs `--privileged` + host cgroups — which dissolves the very fence the container exists to provide on a shared box. Rejected as a default.
assert it at the af layer instead	does not help: with no service manager, `af daemon install` fails outright, so there is no unit to be the supervisor of. The layer was never the problem.
the CI runner	works — it is itself a clean, ephemeral machine with a real systemd. This is where it runs.

Two guards keep that honest:

a SKIP is not a pass: any skipped check exits the run non-zero unless AF_LIFECYCLE_ALLOW_PARTIAL=1 is set. make lifecycle-container sets it (a dev-box container genuinely cannot host a service manager) and prints a PARTIAL COVERAGE banner naming what went untested. The native CI leg also permits an explicit external-release-availability skip; its dedicated check below still makes assertion #4 non-skippable;
the CI native leg requires all three positive assertion-4 PASS records. It fails the job if assertion #4 skips or scenario B stops before reaching it — a green run that quietly stopped testing supervision is the exact lie this gate exists to prevent.

The audit: "what would make this pass without testing anything?"¶

Asked of every assertion in the script. It is the only question that matters for a gate like this, and it found five real holes — four of which would have shipped green:

assertion	what would make it pass vacuously	what stops it now
TUI renders on a virgin home	the doctor probe ran first and created the home + log, so "virgin" was already false when measured. An observation that creates the state it observes is not an observation.	each probe gets its own pristine home and calls `lc_assert_virgin` immediately before touching it — which holds regardless of which command materializes what, on either side of the upgrade
no panic on the first frame	an empty capture: `grep` finds no panic in a dead pane and passes	require the screen to contain the TUI marker before concluding anything about what is not on it (it reports how many lines it searched)
first session created	`af sessions create` answering 0 while its JSON body carries a refusal — observed on this harness	assert the outcome: the daemon must list the session
no version skew (the #1921 check)	`lc_assert_eq "" ""` — if the daemon's version came back empty (it never answered and the `/proc/<pid>/exe` fallback were unreadable) and the client version came back empty, the most important assertion here would compare nothing to nothing and report "no skew"	both operands must be non-empty or it fails
no session Lost	testing `status == 4` (`Dead`) — write-never since #1108, so it matched nothing on any machine	fail-closed: "is every session in a state I recognise as healthy?", so unknown/drifted values fail loudly
unit installed	`af daemon install` returning 0 without registering anything	confirm `daemon status --json .autostart_unit` is true
fault injections	the injection silently no-ops (a CLI change, an RPC error) and the run reports on a scenario that never existed	each injection asserts its outcome (not its exit code) and aborts the scenario if it did not apply
sessions survive	counting 0 before and 0 after (`0 == 0`)	scenario B aborts unless exactly 2 sessions exist first
session counts	an unscoped `sessions list` counting the wrong project	every query passes `--repo`

The general rule this settles on: prefer fail-closed. Enumerating bad states fails open — anything you did not think of passes. Asking "is this provably the good state?" turns every surprise into a loud failure instead of a quiet green.

The harness's own tests (`make lifecycle-selftest`)¶

The gate proves things about af; this proves things about the gate — and it is the cheaper half: pure logic, no containers, no daemons, no network, so it runs on the host in a second and gates every PR.

It covers the three ways this harness can become a rubber stamp, none of which is visible in a passing run:

a fault injection that cannot execute. An unregistered name (skip-restart vs skip-daemon-restart) matches no branch, so nothing is injected and every assertion passes — a green check for an experiment that never happened. Names are now validated against a registry before anything runs (hard exit 2), and execution is recorded: an injection requested but never reached (e.g. asked for scenario B while only scenario A runs) FAILS the run. Never a skip — a skip is how this class hides.
the disposable guard saying yes to a real machine. Detection is positive only and the default is NO: docker (/.dockerenv), podman (/run/.containerenv — testbox supports both engines, so a podman box could not run this gate at all before), or CI=true. An unrecognized runtime is never "probably fine".
concurrent runs sharing an image tag. That is #1171 (the fixed af-playtest name), fixed in #1166 with per-run-unique names; the tag now carries the same per-run token the container does and is removed on exit.

Each case was watched failing against the pre-fix code. That discipline paid for itself immediately: the first version of the podman test re-implemented the detection and grepped the source for /run/.containerenv — and passed against code with podman detection deleted, because that string also appears in the guard's error message. The markers are now injectable variables so the test drives the real function. A test written to confirm a fix is not a test until you have seen it fail.

What this does NOT cover yet¶

macOS / launchd. lc_unit_active and lc_unit_main_pid already have Darwin branches, so assertion #4 asserts the same property against launchd (state = running and pid = N in the gui/<uid> domain af restarts) rather than a re-invented one. Assertion 2 is no longer the blocker — it asks the daemon for its version and only falls back to /proc/<pid>/exe. What still blocks the macos-latest leg is daemon discovery: lc_daemon_pids identifies a daemon by reading /proc/<pid>/cmdline and /proc/<pid>/environ (the check that proves a pid serves this throwaway home before the harness signals it), and assertions 1 and 3 plus teardown are built on it. Porting that scan is the work. Given #1931 turned on macOS CI and immediately found three real darwin defects (#1939, #1940, #1941), this leg is worth landing soon.
#1916's reset-vs-relaunch race — this scenario upgrades, it does not reset.
Downgrades and channel switches (--allow-downgrade, preview → stable).
Package-manager installs (Homebrew, install.sh); the gate installs the release tarball the way af upgrade does.