docs(roadmap): add Pillar 9 — live VM runtime QA
All checks were successful
Check / eval-and-lint (push) Successful in 7m3s

New pillar to drive the distro running in a VM and verify what actually
works vs. what only looks right in source: run every nomarchy-* command,
walk every menu entry, switch through all 22 palettes and eyeball
rendering, fire every keybinding, drive the installer. Documents the
method (build -> boot headless -> screendump/sendkey) proven in the first
VM-boot pass, the per-component checklist, and the deliverable format. The
old "Pillar 8 runtime verification" Now-board punch-list folds into it.
AGENT.md's verify step now points at the headless VM screenshot flow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernardo Magri
2026-05-30 21:58:27 +01:00
parent 70d0b56259
commit 8e2c27087f
2 changed files with 34 additions and 3 deletions

View File

@@ -156,7 +156,7 @@ Steps you should follow for any non-trivial change:
```bash
sudo /etc/install.sh --dry-run # in the live ISO or VM
```
For waybar / Hyprland visual changes, the only reliable check is booting the live ISO with `nomarchy-test-live-iso`. If you can't boot it, **say so** rather than claiming success.
For waybar / Hyprland visual changes, boot it and look. On a host with KVM you can do this headless: `nix build .#nixosConfigurations.default.config.system.build.vm`, run its `bin/run-*-vm` with `QEMU_OPTS="-display none -monitor unix:/tmp/mon.sock,server,nowait -vga none"`, then `echo "screendump /tmp/s.ppm" | socat - unix-connect:/tmp/mon.sock`, convert PPM→PNG, and inspect it (this is how the first Pillar 9 pass caught the boot-time Hyprland config-error overlay). The `nomarchy-test-live-iso` / `installerVm` flows are the ISO-side equivalents. If you genuinely can't boot anything, **say so** rather than claiming a visual change works. See ROADMAP §9 (Live VM runtime QA) for the full method.
6. **Commit narrowly.** One concept per commit. The commit subject is `<type>: <imperative summary>` (`feat:`, `fix:`, `docs:`, `chore:`). The body explains the why.
7. **Push only when the user asks.** Local commits are free; pushing publishes.

View File

@@ -19,7 +19,7 @@ Guardrails (apply when adding anything):
### Now (ready to pick up)
- **Pillar 8 runtime verification pass.** Every Component 110 code sweep has shipped, but each closeout entry flagged behaviour that can only be confirmed on a real install: boot the live ISO and eyeball waybar across panel positions × form factors × all 22 palettes; run every `nomarchy-menu` entry and confirm it does the thing; walk every `nomarchy.*` option's enable→rebuild→observe loop. Run the punch-list, fix any new bugs inline, log structural ones as new roadmap rows.
- **Pillar 9: live VM runtime QA.** Drive the distro running in a VM and check what actually works versus what only looks right in source. The first VM-boot smoke pass shipped (2026-05-30 — found + fixed four runtime bugs invisible to eval/CI). Continue per component: run every `nomarchy-*` command, walk every `nomarchy-menu` entry, switch through all 22 palettes and eyeball rendering, fire every keybinding, drive the installer. See **§9** for the method (build → boot headless → `screendump`/`sendkey`) and the full component checklist. This absorbs the old "Pillar 8 runtime verification" punch-list.
### Next (bigger lifts that build on Now)
@@ -135,7 +135,38 @@ Per-PR deliverable:
Pillar is **done** when every component has a closed `wave/qa-<component>` PR and the roadmap captures every deferred finding.
## 9. Process notes
## 9. Pillar: Live VM runtime QA
Pillar 8 audited feature *code* (read-and-reason) and shipped a first VM-boot smoke pass. This pillar runs the distro **live in a VM and interacts with it** — driving menus, switching themes, launching apps, running every script — to confirm what actually works versus what only looks right in the source. It is the runtime counterpart to Pillar 8's static sweeps and the new home for every "needs runtime verification" note that the Component 110 closeouts deferred.
### Method (how to drive the VM)
- **Build + boot.** `nix build .#nixosConfigurations.default.config.system.build.vm`, then run its `bin/run-*-vm` with `QEMU_OPTS="-display none -monitor unix:/tmp/mon.sock,server,nowait -vga none"`. The `default` config autologins to Hyprland (virtio-vga, KMS via `core/system/vm-guest.nix`).
- **Observe.** Capture the framebuffer with the QEMU monitor's `screendump /path.ppm` over the monitor socket (`echo "screendump …" | socat - unix-connect:/tmp/mon.sock`), convert PPM→PNG, and eyeball. Crop regions for legibility. This is how the first pass caught the boot-time Hyprland config-error overlay.
- **Drive.** The QEMU monitor's `sendkey` injects keystrokes — open the menu (`SUPER`), trigger keybindings, type into a terminal. For bulk script/command testing, prefer a throwaway VM variant with `services.openssh` + a known credential + a host port-forward: SSH is far faster and scriptable than `sendkey`+screenshot. Add it to a test overlay, never to the shipped config.
- **State.** Delete the VM's `*.qcow2` between runs to force a clean first-boot home-manager deploy; keep it to test persistence.
### Components (each is one batch — "do it, then look")
1. **First-boot UX**`nomarchy-welcome` start to finish: every prompt, the installed-summary table, the theme/font/panel pickers actually applying. (First pass fixed the `compgen` bug here.)
2. **Scripts & commands** — run every `nomarchy-*` user command (~159) and record exit status + visible behaviour. Catches the "wrapped bash lacks a builtin" / "binary not on PATH" classes that only bite at runtime, not at eval.
3. **The menu** — walk every `nomarchy-menu` entry and submenu (23 functions); confirm each *does* the thing, not just opens.
4. **Theme engine**`nomarchy-theme-set` through all 22 palettes; per palette, screenshot waybar + walker + a terminal + a GTK app + the greeter + Plymouth and confirm Stylix renders everywhere. Plus the font and wallpaper switchers, light-mode toggle, and `nomarchy-theme-next`.
5. **Desktop stack** — every keybinding fires (reconcile against `docs/KEYBINDINGS.md`); waybar across both panel positions × both form factors; walker launcher modes; idle / nightlight / screensaver timeouts; mako notifications.
6. **Apps** — launch each `features/apps/*` app; confirm it opens and is themed.
7. **Options loop** — for each `nomarchy.*` option: set → rebuild → observe the claimed effect → unset → observe it's gone. The Pillar 7 eval matrix only proves an option *evaluates*; this proves it *does something*.
8. **Installer** — boot `installerVm`; drive `install.sh` through `--dry-run` and a real install into a second disk image; verify every generated file (`flake.nix`, `system.nix`, `home.nix`, `hardware-selection.nix`, `state.json`).
9. **Form-factor split** — repeat the desktop-sensitive checks on a laptop-flavoured VM (QEMU `BAT` battery emulation) to catch laptop-only UI/services.
### Deliverable
- Per batch: a short report — what was run, what worked, what broke, what was fixed inline (with screenshot/log evidence where it matters), what was deferred as a new row.
- Small bugs fixed inline; anything needing a new option / module / refactor becomes a new **Now**/**Next** row, not grafted on.
- Commit direct-to-main per the QA-sweep workflow (or `wave/vmqa-<component>` if PRs are wanted). Doc updates ride with the change per `docs/AGENT.md` §5.4.
Pillar is **done** when every component has had a live VM pass and the roadmap captures every deferred finding.
## 10. Process notes
- **Branch naming:** `wave/<pillar>-<short-slug>`. Examples: `wave/audit-pkg-scripts`, `wave/installer-disk-metadata`, `wave/laptop-preset`.
- **One PR per audit batch.** Reference rows in `docs/SCRIPTS.md`. Smaller PRs review faster.