From 8547fa70d9e5f86e7a35a75fecf9ed7018957096 Mon Sep 17 00:00:00 2001 From: Bernardo Magri Date: Fri, 29 May 2026 21:21:28 +0100 Subject: [PATCH] feat(ci): eval matrix for opt-in nomarchy.* toggles MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit nix flake check --no-build only evaluates the four default configs, so a bug that only fires when a nomarchy.* toggle is flipped sails through — exactly how the vscode option rename and the impermanence systemd-stage-1 assertion both reached main undetected. Add bin/utils/nomarchy-eval-matrix: layers each opt-in scenario (16 toggles, both system and home halves) onto nixosConfigurations.default via extendModules and forces system.build.toplevel.drvPath (which forces assertion checks); also forces all 21 palettes through the system-side Plymouth base00->RGB math that the home-only allThemeVariants never reaches, plus both standalone homeConfigurations. Per-scenario results come from builtins.tryEval so one failure doesn't mask the rest; verified it goes red on a re-injected assertion failure and a bad option ref, green on the current tree. Wired into .forgejo/workflows/check.yml after flake check (jq via nix shell so it doesn't depend on the runner image). AGENT.md §5 now tells future agents to add a scenario when they add/rename an opt-in option. nomarchy-docs-scripts: filter systemd-service declarations (.services. nomarchy-) and denylist the eval-matrix repo tool so neither shows as a dangling "missing" reference; also retroactively clears the latent nomarchy-flathub-init drift from the gaming-flathub commit. Co-Authored-By: Claude Opus 4.8 --- .forgejo/workflows/check.yml | 11 +++ bin/utils/nomarchy-docs-scripts | 5 ++ bin/utils/nomarchy-eval-matrix | 119 ++++++++++++++++++++++++++++++++ docs/AGENT.md | 5 ++ docs/ROADMAP.md | 2 + 5 files changed, 142 insertions(+) create mode 100755 bin/utils/nomarchy-eval-matrix diff --git a/.forgejo/workflows/check.yml b/.forgejo/workflows/check.yml index 6af8bcb..1ec63f2 100644 --- a/.forgejo/workflows/check.yml +++ b/.forgejo/workflows/check.yml @@ -7,6 +7,11 @@ # 3. `docs/SCRIPTS.md` drifts from the repo state because somebody # added / removed / renamed a script and didn't run the generator # (the pre-commit hook handles this, but only when enabled per-clone). +# 4. An opt-in nomarchy.* toggle stops evaluating. `nix flake check` +# only evaluates the four default configs, so a bug that only fires +# when a toggle is flipped sails through (e.g. the impermanence +# systemd-stage-1 assertion). nomarchy-eval-matrix layers each toggle +# onto the default config and forces system.build.toplevel. # # Doesn't build ISOs — that needs a binary cache. Add a separate job # once Cachix/Attic is in place. @@ -36,6 +41,12 @@ jobs: - name: nix flake check --no-build run: nix flake check --no-build + - name: Evaluate opt-in toggle / palette / home matrix + # flake check never flips a nomarchy.* toggle; this does. jq is + # provided via nix shell so the step doesn't depend on the runner + # image preinstalling it. + run: nix shell nixpkgs#jq --command ./bin/utils/nomarchy-eval-matrix + - name: Lint nomarchy-* scripts (bash -n + shellcheck) run: | # Mirror what .githooks/pre-commit runs locally, but across the diff --git a/bin/utils/nomarchy-docs-scripts b/bin/utils/nomarchy-docs-scripts index 6c64f53..9f03f1a 100755 --- a/bin/utils/nomarchy-docs-scripts +++ b/bin/utils/nomarchy-docs-scripts @@ -98,10 +98,12 @@ all_refs=$(grep -rhE 'nomarchy-[a-z0-9]([a-z0-9-]*[a-z0-9])?' \ -e '\./result/bin/run-nomarchy-' \ -e 'mktemp[[:space:]]+[^|]*-t[[:space:]]+nomarchy-' \ -e '(TIMER_NAME|NOPASSWD_FILE|UNIT_NAME)=.*nomarchy-' \ + -e '\.services\.nomarchy-' \ -e 'docker[[:space:]]+[^|]*nomarchy-' \ | grep -oE 'nomarchy-[a-z0-9]([a-z0-9-]*[a-z0-9])?' \ | grep -vE '^(nomarchy-launch|nomarchy-brightness|nomarchy-cmd|nomarchy-pkg|nomarchy-restart|nomarchy-toggle|nomarchy-theme|nomarchy-webapp-handler|nomarchy-font-selector|nomarchy-theme-selector|nomarchy-wallpaper-selector|nomarchy-setup|nomarchy-refresh|nomarchy-scripts|nomarchy-system-scripts|nomarchy-theme-engine-scripts)$' \ | grep -vE '^(nomarchy-plymouth|nomarchy-sddm-theme|nomarchy-live|nomarchy-rev|nomarchy-windows)$' \ + | grep -vE '^(nomarchy-eval-matrix)$' \ | sort -u) # The second denylist covers identifiers whose ambiguity survives the line # filter: `nomarchy-plymouth` / `nomarchy-sddm-theme` are Nix derivation @@ -109,6 +111,9 @@ all_refs=$(grep -rhE 'nomarchy-[a-z0-9]([a-z0-9-]*[a-z0-9])?' \ # ISO label that shows up in comments, `nomarchy-rev` is `/etc/nomarchy-rev` # (written by the ISO), and `nomarchy-windows` is a docker container name # in compose heredocs. +# The third denylist: `nomarchy-eval-matrix` is a bin/utils repo tool (like +# this generator) that names itself in its own header comment — not a +# user-facing script that ships in nomarchy-system-scripts. # --- Render: header -------------------------------------------------------- diff --git a/bin/utils/nomarchy-eval-matrix b/bin/utils/nomarchy-eval-matrix new file mode 100755 index 0000000..94afd38 --- /dev/null +++ b/bin/utils/nomarchy-eval-matrix @@ -0,0 +1,119 @@ +#!/usr/bin/env bash +# nomarchy-eval-matrix +# +# Evaluates the opt-in nomarchy.* surface that `nix flake check` never +# touches. flake check only evaluates the four default configs, so a bug +# that only fires when a toggle is flipped — a renamed option, a failed +# assertion, a stale reference — sails straight through. Two such bugs +# shipped to main before this existed (the vscode option rename and the +# impermanence systemd-stage-1 assertion); both would have been caught +# here. +# +# Each toggle scenario is layered onto nixosConfigurations.default via +# extendModules and its system.build.toplevel.drvPath is forced (which +# also forces the assertion checks). Every palette is forced through the +# system-side Plymouth colour math (which allThemeVariants, being +# home-only, doesn't exercise). Both standalone homeConfigurations are +# forced too. +# +# Per-scenario results come from builtins.tryEval so one failure doesn't +# mask the rest — the whole matrix runs, then the script exits non-zero +# if anything failed. +# +# nomarchy-eval-matrix # run the full matrix, table + exit code +# +# Add a scenario by editing the `toggleScenarios` attrset in the embedded +# Nix expression below. + +set -u + +repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd)" +cd "$repo_root" + +NIX=(nix --extra-experimental-features 'nix-command flakes') + +read -r -d '' EXPR <<'NIXEOF' || true +let + f = builtins.getFlake (toString ./.); + lib = f.inputs.nixpkgs.lib; + themeNames = (import ./lib { inherit lib; }).themeNames; + default = f.nixosConfigurations.default; + + # Opt-in toggles layered onto the default config. Home-side options go + # under home-manager.users.nomarchy.nomarchy.*; system-side directly. + toggleScenarios = { + "impermanence-single" = { nomarchy.system.impermanence.enable = true; }; + "impermanence-multi" = { nomarchy.system.impermanence = { enable = true; mainLuksName = "crypted_main"; }; }; + "laptop-preset" = { nomarchy.system = { formFactor = "laptop"; laptop.enable = true; }; }; + "desktop-preset" = { nomarchy.system = { formFactor = "desktop"; desktop.enable = true; }; }; + "accessibility-system" = { nomarchy.system.accessibility.enable = true; }; + "gaming-system" = { nomarchy.system.gaming.enable = true; }; + "hybrid-gpu" = { nomarchy.system.features.hybridGPU = true; }; + "hibernation" = { nomarchy.system.hibernation.enable = true; }; + "docker" = { nomarchy.system.virtualization.docker.enable = true; }; + "fwupd" = { nomarchy.hardware.fwupd = true; }; + "accessibility-home" = { home-manager.users.nomarchy.nomarchy.accessibility.enable = true; }; + "gaming-home" = { home-manager.users.nomarchy.nomarchy.gaming.enable = true; }; + "overrides" = { home-manager.users.nomarchy.nomarchy.overrides.enable = true; }; + "panel-bottom" = { home-manager.users.nomarchy.nomarchy.panelPosition = "bottom"; }; + "keymap-variant" = { home-manager.users.nomarchy.nomarchy.keymap = { layout = "de"; variant = "nodeadkeys"; }; }; + "toggles-off" = { home-manager.users.nomarchy.nomarchy.toggles = { waybar = false; idle = false; nightlight = false; }; }; + }; + + forced = x: (builtins.tryEval (builtins.seq x true)).success; + + tryToplevel = mod: + forced (default.extendModules { modules = [ mod ]; }).config.system.build.toplevel.drvPath; + + # Palettes only need the system-side Plymouth path forced — that's the + # per-palette base00 → RGB float math in themes/engine/plymouth.nix that + # the home-only allThemeVariants never reaches. + tryPalette = name: + forced (default.extendModules { modules = [ { nomarchy.system.theme = name; } ]; }).config.boot.plymouth.themePackages; + + tryHome = name: forced f.homeConfigurations.${name}.activationPackage.drvPath; +in { + toggles = builtins.mapAttrs (_: tryToplevel) toggleScenarios; + palettes = builtins.listToAttrs (map (n: { name = n; value = tryPalette n; }) themeNames); + home = builtins.listToAttrs (map (n: { name = n; value = tryHome n; }) [ "nomarchy" "nixos" ]); +} +NIXEOF + +echo "Evaluating opt-in toggle + palette + home matrix (this takes a few minutes)..." +json="$("${NIX[@]}" eval --impure --json --expr "$EXPR" 2>/tmp/eval-matrix.err)" +status=$? +if [[ $status -ne 0 || -z "$json" ]]; then + echo "ERROR: the matrix evaluation aborted before producing results." >&2 + echo "This usually means an uncatchable error (abort/infinite recursion) in one" >&2 + echo "scenario, or a syntax error in the expression. Full output:" >&2 + cat /tmp/eval-matrix.err >&2 + exit 1 +fi + +# Render each section and tally failures. +fails=0 +render() { + local section="$1" + echo + echo "=== $section ===" + while IFS=$'\t' read -r name ok; do + if [[ "$ok" == "true" ]]; then + printf ' ok %s\n' "$name" + else + printf ' FAIL %s\n' "$name" + fails=$((fails + 1)) + fi + done < <(echo "$json" | jq -r --arg s "$section" '.[$s] | to_entries[] | "\(.key)\t\(.value)"' | sort) +} + +render toggles +render palettes +render home + +echo +if [[ $fails -gt 0 ]]; then + echo "$fails scenario(s) failed to evaluate. Re-run the failing one directly for the trace, e.g.:" + echo " nix eval --impure --show-trace --expr 'let f = builtins.getFlake (toString ./.); in (f.nixosConfigurations.default.extendModules { modules = [ { } ]; }).config.system.build.toplevel.drvPath'" + exit 1 +fi +echo "All scenarios evaluated cleanly." diff --git a/docs/AGENT.md b/docs/AGENT.md index 92e3490..ce93bd4 100644 --- a/docs/AGENT.md +++ b/docs/AGENT.md @@ -143,6 +143,11 @@ Steps you should follow for any non-trivial change: nix --extra-experimental-features 'nix-command flakes' flake check --no-build bash -n installer/install.sh # if you touched it ``` + `flake check` only evaluates the four *default* configs — it never flips a `nomarchy.*` toggle. If you touched anything that only fires when a toggle is enabled (impermanence, a preset, an opt-in feature, the per-palette system theme), also run the toggle matrix, which layers each opt-in scenario onto the default config and forces it: + ```bash + ./bin/utils/nomarchy-eval-matrix + ``` + When you **add or rename an opt-in `nomarchy.*` option**, add a matching scenario to the `toggleScenarios` attrset in that script — otherwise the new surface is unverified and the next eval-time bug in it ships silently (this is exactly how the impermanence assertion and the vscode option rename reached `main`). The matrix also runs in CI (`.forgejo/workflows/check.yml`). First clone? Enable the repo's pre-commit hook so `docs/SCRIPTS.md` regenerates whenever you add/modify/remove a `nomarchy-*` script: ```bash git config core.hooksPath .githooks diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 06635b3..6043e38 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -147,6 +147,8 @@ Pillar is **done** when every component has a closed `wave/qa-` PR an (Move items here when they land — keep them brief, link the commit/PR.) +- _2026-05-29_ — **Pillar 7: opt-in toggle eval matrix in CI.** `nix flake check --no-build` only evaluates the four default `nixosConfigurations`/`homeConfigurations`, so any eval-time bug that *only* fires when a `nomarchy.*` toggle is flipped sailed straight through — exactly how the vscode option rename and the impermanence systemd-stage-1 assertion both reached `main` undetected. New `bin/utils/nomarchy-eval-matrix` layers each opt-in scenario (16 toggles: impermanence single/multi, laptop/desktop presets, accessibility, gaming, hybridGPU, hibernation, docker, fwupd, overrides, panel position, keymap variant, toggles-off — plus home + system halves) onto `nixosConfigurations.default` via `extendModules` and forces `system.build.toplevel.drvPath` (which forces the assertion checks); it also forces all 21 palettes through the system-side Plymouth `base00`→RGB-float math that the home-only `allThemeVariants` never reaches, and both standalone `homeConfigurations`. Per-scenario results come from `builtins.tryEval` so one failure doesn't mask the rest — verified the harness goes red on both a failed assertion (re-injected `boot.initrd.postDeviceCommands`) and a nonexistent-option ref, and green on the current tree (all 39 scenarios pass). Wired into `.forgejo/workflows/check.yml` as a step after `flake check`, with `jq` supplied via `nix shell` so it doesn't depend on the runner image. `docs/AGENT.md` §5.5 now instructs: when you add or rename an opt-in `nomarchy.*` option, add a matching scenario to the script's `toggleScenarios` attrset. + - _2026-05-29_ — **Impermanence rollback no longer fails to build.** `core/system/impermanence.nix` implemented the Erase-Your-Darlings root wipe via `boot.initrd.postDeviceCommands`, but `themes/engine/plymouth.nix:63` sets `boot.initrd.systemd.enable = true` distro-wide (Plymouth is imported unconditionally from `core/system/default.nix`), and systemd stage-1 initrd hard-rejects `postDeviceCommands` with a failed assertion. Net effect: **every** install that flipped `nomarchy.system.impermanence.enable = true` — including any user who picked impermanence at the installer prompt — produced a config that wouldn't evaluate, let alone boot. Converted the rollback into a `boot.initrd.systemd.services.nomarchy-rollback` oneshot unit ordered `after = systemd-cryptsetup@${mainLuksName}.service` and `before = sysroot.mount`, with `boot.initrd.systemd.initrdBin` pulling in `btrfs-progs` / `coreutils` / `util-linux` / `findutils` (the systemd initrd doesn't ship them by default, unlike the old scripted initrd). Script body (timestamped `@` → `old_roots` move, 30-day recursive subvolume GC, `root-blank` → `@` snapshot) is unchanged. Verified the assertion is gone and `nixosConfigurations.default` + `impermanence.enable` evaluates fully via `extendModules`. **Runtime-verification caveat:** the boot-time wipe semantics (the `[[ ]]`/function/IFS shell idioms and the cleanup loop under the systemd initrd shell, plus correct ordering vs the LUKS mapping) can only be confirmed by a real wipe-boot cycle on an impermanence install — fold into the Pillar 8 punch-list. `docs/TROUBLESHOOTING.md` persistence-block line reference updated (46-72 → 91-120). - _2026-05-22_ — **VSCode theme extensions pinned for 10 marketplace palettes.** Before this fix, only the 6 palettes whose theme extensions ship in `pkgs.vscode-extensions` (catppuccin, catppuccin-latte, nord, tokyo-night, rose-pine, gruvbox) had working VSCode theming — every other palette had `workbench.colorTheme` set to a theme VSCode couldn't find, so it silently fell back to the built-in default. Including the default `summer-night` palette (sainnhe.everforest), which meant the default install had broken VSCode theming. Probed all 13 unique extensions against the VSCode marketplace extensionquery API; 10 exist (`sainnhe.everforest`, `shadesOfBuntu.flexoki-light`, `qufiwefefwoyn.kanagawa`, `oldjobobo.{lumon,miasma,retro-82}-theme`, `TahaYVR.matteblack`, `jovejonovski.ocean-green`, `monokai.theme-monokai-pro-vscode`, `Bjarne.white-theme`) and 3 don't (`Bjarne.{ethereal,hackerman,vantablack}-nomarchy` — unpublished custom Nomarchy themes; logged as a new Later row). Fetched the latest version + sha256 for each of the 10 via `nix store prefetch-file` against `https://${publisher}.gallery.vsassets.io/_apis/public/gallery/publisher/${publisher}/extension/${name}/${version}/assetbyname/Microsoft.VisualStudio.Services.VSIXPackage`. Added a `marketplaceExtensions` list to `features/apps/vscode.nix` that wraps each entry in `pkgs.vscode-utils.extensionFromVscodeMarketplace { publisher; name; version; sha256; }` and concatenates with the existing nixpkgs-packaged list. Smoke-built `sainnhe.everforest` end-to-end to confirm the URL pattern + sha256 resolve. Documented the version-bump procedure in the module comment. `docs/OPTIONS.md` updated to drop the "still break" caveat for everything except the three Bjarne palettes.