Bernardo Magri
07400ab4f9
All checks were successful
Check / eval (push) Successful in 4m16s
Nomarchy now carries a nixpkgs-unstable input and exposes it through overlays.default as an `unstable` attrset: in the template's `home.packages = with pkgs; [ … ]` a user writes `unstable.lmstudio` and is done. Downstream stays one locked input; nomarchy-pull moves it. lib.nix needed zero changes — mkFlake's pkgs already applies the overlay. The import is plain and non-recursive, so configs that never reference unstable. pay nothing beyond the input fetch; allowUnfree mirrors the base set so the motivating lmstudio resolves. Home packages are the supported surface (documented convention). Costs stated in the README, not hidden: you own the pinned+unstable combination; a second toolchain lands in your closure (~690 of ~4k paths shared, measured on lmstudio); it moves when the lock moves. Lock bumps now move two channels — VERIFICATION §5 and CONVENTIONS updated accordingly. flake.lock gained exactly one node (nixos-unstable @ 753cc8a); nothing else moved. Verification: V1 — nix flake check --no-build green; new eval-only checks.unstable-seam green (26.05pre vs 26.11pre lib.version, no builds); template HM activation built via mkFlake with unstable.hello enabled and its closure carries the unstable store path (pg2zfr… vs pinned nm7p8w…). Implementation by a Sonnet subagent; design and review on Fable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
182 lines
9.1 KiB
Markdown
182 lines
9.1 KiB
Markdown
# Verification — the enforcement rules
|
||
|
||
Nomarchy's promise to its user: a rock-stable, fully functional, beautiful
|
||
workstation that is reproducible, easy to recover, and never requires the
|
||
user to become a Nix expert. Every rule below exists to protect that
|
||
promise. A change that works but degrades stability, aesthetics, or
|
||
user-simplicity is a regression, not a feature.
|
||
|
||
This document is an **enforcer**, not the workflow itself: the ladder and
|
||
iteration protocol live in `agent/LOOP.md`, VM instructions and gotchas in
|
||
`docs/TESTING.md`. It applies to **every** change to this repo — features,
|
||
fixes, theming, module changes, lock bumps, docs, backlog grooming.
|
||
"Small" or "obvious" changes are exactly where verification gets skipped,
|
||
so they trigger it too. If this file and those docs ever disagree, fix the
|
||
discrepancy in the same or a follow-up commit so they can't disagree
|
||
twice.
|
||
|
||
## 1. Preflight (once per session)
|
||
|
||
Before starting work, establish what verification tier this environment
|
||
can reach, so you never promise verification you can't deliver:
|
||
|
||
1. Linux x86_64 host? `/dev/kvm` present and readable?
|
||
2. Enough free disk for an image/ISO build (multi-GB)?
|
||
3. Network access for a cold Nix store?
|
||
|
||
If the environment cannot reach V2 (no KVM, no disk, etc.): say so
|
||
immediately, do the V0/V1 work honestly, mark the change **"V2 pending"**
|
||
exactly as you would mark a hardware-blocked change "V3 pending" (§4),
|
||
and stop short of claiming the change is done. Never simulate, guess, or
|
||
describe what a VM test "would" show.
|
||
|
||
## 2. The verification ladder (enforcement rules)
|
||
|
||
Climb the V0–V3 ladder as defined in `agent/LOOP.md`. Four non-negotiable
|
||
enforcement rules on top:
|
||
|
||
1. **V2 is mandatory for anything user-visible.** If a user of the
|
||
installed system could perceive the change — behavior, layout, colors,
|
||
keybinds, timing, error messages — it must be exercised in the local
|
||
VM before commit. Docs-only, comment-only, or agent-notes changes may
|
||
stop at the tier LOOP.md assigns them; user-visible changes may not.
|
||
2. **Every "done" report names the tier reached and shows the evidence.**
|
||
Evidence means: the command run and its relevant output, the checklist
|
||
items exercised, and for visual work the screenshots viewed (§3). "It
|
||
builds" is a V1 claim, not a V2 claim. Never let a report imply a
|
||
higher tier than was actually reached.
|
||
3. **A failed or flaky test is a result, not an obstacle.** Distinguish
|
||
real failures from environment flakes using the known-gotchas section
|
||
of `docs/TESTING.md` (e.g. no-KVM slowness, missing guest GL). If you
|
||
cannot confidently classify a failure, report it as unresolved — do
|
||
not retry until green and report only the green run.
|
||
4. **VM runs are headless and unattended.** Use the repo's headless
|
||
harness — `tools/test-live-iso.sh` and `tools/test-install.sh` for
|
||
boot/install runs, `tools/vm/qmp.py` for programmatic VM control and
|
||
`tools/vm/vncshot.py` for screen capture — never a graphical VM window
|
||
or any flow that needs a human at the console. The human is not part
|
||
of the test loop: do not pause mid-run to ask them to look at the VM,
|
||
click something, or confirm what is on screen. A run must complete on
|
||
its own and leave auditable artifacts behind (logs, serial console
|
||
output, exit codes, screenshots), with every wait bounded by a timeout
|
||
so a hang becomes a recorded failure instead of a stalled session.
|
||
Prefer scripted assertions (process up, file exists, service/D-Bus
|
||
state, the checks in `tools/`) over eyeballing; where judgment is
|
||
genuinely needed — visual quality — *you* view the captured
|
||
screenshots (§3), not the human. The human reviews evidence in the
|
||
final report, never the live run.
|
||
|
||
### Regression scope after a change
|
||
|
||
Re-running the full checklist for every change wastes VM time; running
|
||
nothing invites regressions. Default rule:
|
||
|
||
- Always: the session-sanity items (boot to session, bar renders).
|
||
- Plus: every checklist item touching the layer you changed.
|
||
- Plus: the theming end-to-end item whenever theming plumbing changed,
|
||
even indirectly (palette generation, symlinks, reload hooks).
|
||
- Lock bumps and toolchain changes: run the full checklist — their blast
|
||
radius is unknowable by construction.
|
||
|
||
## 3. Visual verification protocol
|
||
|
||
Visual quality is a core feature of Nomarchy, so "it probably looks fine"
|
||
is never verification. A visual/UI change is not V2-verified until all of
|
||
the following are true:
|
||
|
||
1. **Before/after screenshots** of the changed surface were captured
|
||
headlessly — `tools/theme-shot.nix` for reproducible theme renders,
|
||
`tools/vm/vncshot.py` (driven via `tools/vm/qmp.py`) for captures from
|
||
a running VM. No VM window, no human interaction. Capture the "before"
|
||
from the base branch or prior generation, not from memory.
|
||
2. **Scripted checks first**: run `tools/check-theme-contrast.py` and
|
||
`tools/audit-theme-design.py` against the affected theme(s) before any
|
||
eyeballing — machine-checkable legibility/design violations should
|
||
never survive to the judgment stage. Use `tools/vm/gap-analysis.py`
|
||
where it applies.
|
||
3. **Two themes**: repeat the "after" capture under at least two themes,
|
||
one with a generated palette and one whole-swap theme (e.g.
|
||
summer-night). These exercise different code paths in the bar/launcher
|
||
theming; a change that looks right under one can silently break the
|
||
other.
|
||
4. **You actually viewed the images** — open the screenshot files and
|
||
look at them. State concretely what you inspected: alignment, spacing,
|
||
contrast/legibility against the palette, icon rendering, no clipped or
|
||
overlapping elements, and that the change looks intentional next to
|
||
the "before".
|
||
5. Keep the screenshots in the run's working area and reference their
|
||
paths in the report, so the human can audit the same evidence.
|
||
|
||
If the VM cannot render the surface faithfully (known GL/compositor gaps
|
||
in the guest — see `docs/TESTING.md`), that specific visual aspect is
|
||
hardware-blocked: verify everything the VM *can* show, and queue the rest
|
||
per §4.
|
||
|
||
## 4. Hardware-blocked checks
|
||
|
||
Some checks genuinely require real hardware (GPU behavior, multi-monitor
|
||
hotplug, audio devices, power/suspend, firmware). For those:
|
||
|
||
1. Add an entry to `agent/HARDWARE-QUEUE.md` with: what changed, **exact**
|
||
reproduction steps a human can follow verbatim, the expected
|
||
observation (what "pass" looks like), and the commit hash once known.
|
||
2. Mark the commit body **"V3 pending: <one-line summary>"**.
|
||
3. Say it plainly in your report. A hardware-blocked check is not a
|
||
failure and not something to hide — hiding it is the failure.
|
||
4. When the human reports back, close the queue entry in the next commit
|
||
and record the outcome; if it failed on hardware, that's a new bug at
|
||
the top of the backlog.
|
||
|
||
Do not use the hardware queue as an escape hatch: if a check *can* be
|
||
done in the VM, it must be. "The VM is slow" does not qualify.
|
||
|
||
## 5. Maintenance work
|
||
|
||
Maintenance follows the same ladder:
|
||
|
||
- **Flake lock bumps**: treat as maximum-blast-radius changes. Build,
|
||
boot the VM, run the full regression checklist, and do a visual
|
||
spot-check of the session (themes can shift with upstream package
|
||
changes). Never merge a lock bump on "it evaluates". Since #134 the
|
||
lock carries **two channels** (the release pin and the
|
||
`nixos-unstable` pin feeding `unstable.*`) — a bump moves both, and
|
||
this checklist applies to both.
|
||
- **Theme imports / new themes**: import via `tools/import-palettes.py`,
|
||
then the full §3 visual protocol; additionally verify the theme-switch
|
||
round trip (into the new theme and back out).
|
||
- **Docs drift**: run `tools/check-option-docs.py` after any change that
|
||
adds or modifies options, and fix drift in the same commit as the code
|
||
change that created it. Doc-only fixes are V0 — but verify any command
|
||
you document by actually running it.
|
||
- **Backlog grooming / agent-notes**: V0; keep entries consistent with
|
||
the conventions in `agent/`.
|
||
|
||
## 6. Guarding the philosophy
|
||
|
||
Before committing, check the change against the distro's promises
|
||
(`agent/GOALS.md` is the full statement):
|
||
|
||
- **User is not a Nix expert.** If the change requires the user to write
|
||
or read Nix to use the feature day-to-day, redesign it. Configuration
|
||
the user touches must stay in the simple, documented surface the repo
|
||
defines.
|
||
- **Rock-stable and recoverable.** Prefer boring, reproducible mechanisms
|
||
over clever ones. Any change that could break boot or the session must
|
||
have an obvious rollback story (NixOS generations count, but say so).
|
||
- **Aesthetics are load-bearing.** A functionally correct but visually
|
||
regressive change fails review by definition — that's what §3 is for.
|
||
|
||
When a requested change conflicts with these promises, stop and raise the
|
||
conflict instead of implementing it quietly.
|
||
|
||
## 7. Reporting format
|
||
|
||
End every unit of work with a short report containing:
|
||
|
||
1. What changed (one paragraph, plain language).
|
||
2. Verification tier reached, with evidence (commands + key output,
|
||
checklist items run, screenshot paths viewed).
|
||
3. Anything pending: "V2 pending" (environment) or "V3 pending"
|
||
(hardware, with queue entry reference).
|
||
4. Follow-ups added to the backlog, if any.
|