Nomarchy/agent/VERIFICATION.md

Verification — the enforcement rules

Nomarchy's promise to its user: a rock-stable, fully functional, beautiful workstation that is reproducible, easy to recover, and never requires the user to become a Nix expert. Every rule below exists to protect that promise. A change that works but degrades stability, aesthetics, or user-simplicity is a regression, not a feature.

This document is an enforcer, not the workflow itself: the ladder and iteration protocol live in agent/LOOP.md, VM instructions and gotchas in docs/TESTING.md. It applies to every change to this repo — features, fixes, theming, module changes, lock bumps, docs, backlog grooming. "Small" or "obvious" changes are exactly where verification gets skipped, so they trigger it too. If this file and those docs ever disagree, fix the discrepancy in the same or a follow-up commit so they can't disagree twice.

1. Preflight (once per session)

Before starting work, establish what verification tier this environment can reach, so you never promise verification you can't deliver:

1. Linux x86_64 host? /dev/kvm present and readable?
2. Enough free disk for an image/ISO build (multi-GB)?
3. Network access for a cold Nix store?

If the environment cannot reach V2 (no KVM, no disk, etc.): say so immediately, do the V0/V1 work honestly, mark the change "V2 pending" exactly as you would mark a hardware-blocked change "V3 pending" (§4), and stop short of claiming the change is done. Never simulate, guess, or describe what a VM test "would" show.

2. The verification ladder (enforcement rules)

Climb the V0–V3 ladder as defined in agent/LOOP.md. Four non-negotiable enforcement rules on top:

1. V2 is mandatory for anything user-visible. If a user of the installed system could perceive the change — behavior, layout, colors, keybinds, timing, error messages — it must be exercised in the local VM before commit. Docs-only, comment-only, or agent-notes changes may stop at the tier LOOP.md assigns them; user-visible changes may not.
2. Every "done" report names the tier reached and shows the evidence. Evidence means: the command run and its relevant output, the checklist items exercised, and for visual work the screenshots viewed (§3). "It builds" is a V1 claim, not a V2 claim. Never let a report imply a higher tier than was actually reached.
3. A failed or flaky test is a result, not an obstacle. Distinguish real failures from environment flakes using the known-gotchas section of docs/TESTING.md (e.g. no-KVM slowness, missing guest GL). If you cannot confidently classify a failure, report it as unresolved — do not retry until green and report only the green run.
4. VM runs are headless and unattended. Use the repo's headless harness — tools/test-live-iso.sh and tools/test-install.sh for boot/install runs, tools/vm/qmp.py for programmatic VM control and tools/vm/vncshot.py for screen capture — never a graphical VM window or any flow that needs a human at the console. The human is not part of the test loop: do not pause mid-run to ask them to look at the VM, click something, or confirm what is on screen. A run must complete on its own and leave auditable artifacts behind (logs, serial console output, exit codes, screenshots), with every wait bounded by a timeout so a hang becomes a recorded failure instead of a stalled session. Prefer scripted assertions (process up, file exists, service/D-Bus state, the checks in tools/) over eyeballing; where judgment is genuinely needed — visual quality — you view the captured screenshots (§3), not the human. The human reviews evidence in the final report, never the live run.

Regression scope after a change

Re-running the full checklist for every change wastes VM time; running nothing invites regressions. Default rule:

• Always: the session-sanity items (boot to session, bar renders).
• Plus: every checklist item touching the layer you changed.
• Plus: the theming end-to-end item whenever theming plumbing changed, even indirectly (palette generation, symlinks, reload hooks).
• Lock bumps and toolchain changes: run the full checklist — their blast radius is unknowable by construction.

3. Visual verification protocol

Visual quality is a core feature of Nomarchy, so "it probably looks fine" is never verification. A visual/UI change is not V2-verified until all of the following are true:

1. Before/after screenshots of the changed surface were captured headlessly — tools/theme-shot.nix for reproducible theme renders, tools/vm/vncshot.py (driven via tools/vm/qmp.py) for captures from a running VM. No VM window, no human interaction. Capture the "before" from the base branch or prior generation, not from memory.
2. Scripted checks first: run tools/check-theme-contrast.py and tools/audit-theme-design.py against the affected theme(s) before any eyeballing — machine-checkable legibility/design violations should never survive to the judgment stage. Use tools/vm/gap-analysis.py where it applies.
3. Two themes: repeat the "after" capture under at least two themes, one with a generated palette and one whole-swap theme (e.g. summer-night). These exercise different code paths in the bar/launcher theming; a change that looks right under one can silently break the other.
4. You actually viewed the images — open the screenshot files and look at them. State concretely what you inspected: alignment, spacing, contrast/legibility against the palette, icon rendering, no clipped or overlapping elements, and that the change looks intentional next to the "before".
5. Keep the screenshots in the run's working area and reference their paths in the report, so the human can audit the same evidence.

If the VM cannot render the surface faithfully (known GL/compositor gaps in the guest — see docs/TESTING.md), that specific visual aspect is hardware-blocked: verify everything the VM can show, and queue the rest per §4.

4. Hardware-blocked checks

Some checks genuinely require real hardware (GPU behavior, multi-monitor hotplug, audio devices, power/suspend, firmware). For those:

1. Add an entry to agent/HARDWARE-QUEUE.md with: what changed, exact reproduction steps a human can follow verbatim, the expected observation (what "pass" looks like), and the commit hash once known.
2. Mark the commit body "V3 pending: ".
3. Say it plainly in your report. A hardware-blocked check is not a failure and not something to hide — hiding it is the failure.
4. When the human reports back, close the queue entry in the next commit and record the outcome; if it failed on hardware, that's a new bug at the top of the backlog.

Do not use the hardware queue as an escape hatch: if a check can be done in the VM, it must be. "The VM is slow" does not qualify.

5. Maintenance work

Maintenance follows the same ladder:

• Flake lock bumps: treat as maximum-blast-radius changes. Build, boot the VM, run the full regression checklist, and do a visual spot-check of the session (themes can shift with upstream package changes). Never merge a lock bump on "it evaluates".
• Theme imports / new themes: import via tools/import-palettes.py, then the full §3 visual protocol; additionally verify the theme-switch round trip (into the new theme and back out).
• Docs drift: run tools/check-option-docs.py after any change that adds or modifies options, and fix drift in the same commit as the code change that created it. Doc-only fixes are V0 — but verify any command you document by actually running it.
• Backlog grooming / agent-notes: V0; keep entries consistent with the conventions in agent/.

6. Guarding the philosophy

Before committing, check the change against the distro's promises (agent/GOALS.md is the full statement):

• User is not a Nix expert. If the change requires the user to write or read Nix to use the feature day-to-day, redesign it. Configuration the user touches must stay in the simple, documented surface the repo defines.
• Rock-stable and recoverable. Prefer boring, reproducible mechanisms over clever ones. Any change that could break boot or the session must have an obvious rollback story (NixOS generations count, but say so).
• Aesthetics are load-bearing. A functionally correct but visually regressive change fails review by definition — that's what §3 is for.

When a requested change conflicts with these promises, stop and raise the conflict instead of implementing it quietly.

7. Reporting format

End every unit of work with a short report containing:

1. What changed (one paragraph, plain language).
2. Verification tier reached, with evidence (commands + key output, checklist items run, screenshot paths viewed).
3. Anything pending: "V2 pending" (environment) or "V3 pending" (hardware, with queue entry reference).
4. Follow-ups added to the backlog, if any.