bernardo/Nomarchy
1
1
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ci: collapse eval matrix from ~39 evals to ~3 (was timing out at 3h)
Some checks failed
Check / eval-and-lint (push) Failing after 6m40s
Details

Browse Source 
The per-toggle + per-palette matrix did ~39 full-system evaluations (16
toggle toplevels + 21 palettes each via a full extendModules eval + 2
home) and ran ~2h55m on the small self-hosted runner — long enough to hit
act_runner's 3h container lifetime (sleep 10800) and get killed.

Redesign:
  * Combine compatible toggles into 3 configs (laptop-stack, desktop-stack,
    impermanence-multi) instead of one eval per toggle. A failure means
    re-run the suspect toggle alone; the trace hint covers that.
  * Check palettes directly via the lib — the per-palette risk is the pure
    base00 -> RGB hex math in plymouth.nix, which needs no module-system
    instantiation. Forcing the palette attrset also catches a missing one.
  * Drop the home configs; the per-output flake-check step already forces
    both homeConfigurations.

~39 full evals -> 3, plus a near-instant palette pass. Runs in ~1m locally
(warm). AGENT.md updated for the new combined-scenario structure.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Bernardo Magri
2026-05-30 08:09:23 +01:00
parent 2dda8e2bb5
commit e8cb346759
2 changed files with 82 additions and 52 deletions
Download Patch File Download Diff File Expand all files Collapse all files

132
bin/utils/nomarchy-eval-matrix
View File

@@ -1,29 +1,38 @@
#!/usr/bin/env bash
# nomarchy-eval-matrix
#
# Evaluates the opt-in nomarchy.* surface that `nix flake check` never
# touches. flake check only evaluates the four default configs, so a bug
# Evaluates the opt-in nomarchy.* surface that the per-output flake check
# never touches. flake check only evaluates the default configs, so a bug
# that only fires when a toggle is flipped — a renamed option, a failed
# assertion, a stale reference — sails straight through. Two such bugs
# shipped to main before this existed (the vscode option rename and the
# impermanence systemd-stage-1 assertion); both would have been caught
# here.
#
# Each toggle scenario is layered onto nixosConfigurations.default via
# extendModules and its system.build.toplevel.drvPath is forced (which
# also forces the assertion checks). Every palette is forced through the
# system-side Plymouth colour math (which allThemeVariants, being
# home-only, doesn't exercise). Both standalone homeConfigurations are
# forced too.
# Cost matters: this runs in CI on a small self-hosted runner, and the
# first cut (one full `extendModules` eval per individual toggle, plus one
# per palette) did ~39 whole-system evaluations and ran ~3h. So:
#
# Per-scenario results come from builtins.tryEval so one failure doesn't
# mask the rest — the whole matrix runs, then the script exits non-zero
# if anything failed.
# * Compatible toggles are COMBINED into a few configs (a failure means
# re-run the offending toggle alone to pinpoint it — see the trace hint
# at the end). laptop vs desktop formFactor conflict, so they split;
# impermanence's multi-disk variant gets its own config for the alt
# mainLuksName.
# * Palettes are checked DIRECTLY via the lib — the per-palette risk is
# the system-side Plymouth base00 → RGB math (fromHexString of 2-char
# slices in themes/engine/plymouth.nix), which is pure and needs no
# module-system instantiation. Forcing the whole palette attrset also
# catches a missing/garbled palette.
# * The standalone homeConfigurations aren't re-checked here — the
# per-output flake-check step already forces both.
#
# nomarchy-eval-matrix # run the full matrix, table + exit code
# Net: a handful of full evals instead of ~39. Results come from
# builtins.tryEval so one failure doesn't mask the rest.
#
# Add a scenario by editing the `toggleScenarios` attrset in the embedded
# Nix expression below.
# nomarchy-eval-matrix # run the matrix, table + exit code
#
# Add a new opt-in option's coverage by dropping it into the relevant
# combined scenario in the `scenarios` attrset below.
set -u
@@ -36,50 +45,71 @@ read -r -d '' EXPR <<'NIXEOF' || true
let
f = builtins.getFlake (toString ./.);
lib = f.inputs.nixpkgs.lib;
themeNames = (import ./lib { inherit lib; }).themeNames;
nl = import ./lib { inherit lib; };
default = f.nixosConfigurations.default;
# Opt-in toggles layered onto the default config. Home-side options go
# under home-manager.users.nomarchy.nomarchy.*; system-side directly.
toggleScenarios = {
"impermanence-single" = { nomarchy.system.impermanence.enable = true; };
"impermanence-multi" = { nomarchy.system.impermanence = { enable = true; mainLuksName = "crypted_main"; }; };
"laptop-preset" = { nomarchy.system = { formFactor = "laptop"; laptop.enable = true; }; };
"desktop-preset" = { nomarchy.system = { formFactor = "desktop"; desktop.enable = true; }; };
"accessibility-system" = { nomarchy.system.accessibility.enable = true; };
"gaming-system" = { nomarchy.system.gaming.enable = true; };
"hybrid-gpu" = { nomarchy.system.features.hybridGPU = true; };
"hibernation" = { nomarchy.system.hibernation.enable = true; };
"docker" = { nomarchy.system.virtualization.docker.enable = true; };
"fwupd" = { nomarchy.hardware.fwupd = true; };
"accessibility-home" = { home-manager.users.nomarchy.nomarchy.accessibility.enable = true; };
"gaming-home" = { home-manager.users.nomarchy.nomarchy.gaming.enable = true; };
"overrides" = { home-manager.users.nomarchy.nomarchy.overrides.enable = true; };
"panel-bottom" = { home-manager.users.nomarchy.nomarchy.panelPosition = "bottom"; };
"keymap-variant" = { home-manager.users.nomarchy.nomarchy.keymap = { layout = "de"; variant = "nodeadkeys"; }; };
"toggles-off" = { home-manager.users.nomarchy.nomarchy.toggles = { waybar = false; idle = false; nightlight = false; }; };
};
forced = x: (builtins.tryEval (builtins.seq x true)).success;
tryToplevel = mod:
# Combined opt-in scenarios. Each enables as many compatible toggles as
# possible so CI does ~3 full-system evals, not one per toggle. Home-side
# options go under home-manager.users.nomarchy.nomarchy.*.
scenarios = {
# laptop formFactor + every compatible system/home toggle + impermanence.
"laptop-stack" = {
nomarchy.system = {
formFactor = "laptop";
laptop.enable = true;
accessibility.enable = true;
gaming.enable = true;
features.hybridGPU = true;
hibernation.enable = true;
virtualization.docker.enable = true;
impermanence.enable = true;
};
nomarchy.hardware.fwupd = true;
home-manager.users.nomarchy.nomarchy = {
accessibility.enable = true;
gaming.enable = true;
overrides.enable = true;
panelPosition = "bottom";
keymap = { layout = "de"; variant = "nodeadkeys"; };
toggles = { waybar = false; idle = false; nightlight = false; };
};
};
# desktop preset — conflicts with the laptop formFactor, so its own eval.
"desktop-stack" = {
nomarchy.system = { formFactor = "desktop"; desktop.enable = true; };
};
# impermanence multi-disk variant (alternate mainLuksName).
"impermanence-multi" = {
nomarchy.system.impermanence = { enable = true; mainLuksName = "crypted_main"; };
};
};
evalToplevel = mod:
forced (default.extendModules { modules = [ mod ]; }).config.system.build.toplevel.drvPath;
# Palettes only need the system-side Plymouth path forced — that's the
# per-palette base00 → RGB float math in themes/engine/plymouth.nix that
# the home-only allThemeVariants never reaches.
tryPalette = name:
forced (default.extendModules { modules = [ { nomarchy.system.theme = name; } ]; }).config.boot.plymouth.themePackages;
tryHome = name: forced f.homeConfigurations.${name}.activationPackage.drvPath;
# base00 → the three byte values the Plymouth template substitutes; plus
# the whole palette attrset, so a missing/garbled palette is caught.
paletteOk = name:
let
p = nl.getPalette name;
b = p.base00;
bytes = [
(lib.fromHexString (lib.substring 0 2 b))
(lib.fromHexString (lib.substring 2 2 b))
(lib.fromHexString (lib.substring 4 2 b))
];
in forced (builtins.deepSeq [ p bytes ] true);
in {
toggles = builtins.mapAttrs (_: tryToplevel) toggleScenarios;
palettes = builtins.listToAttrs (map (n: { name = n; value = tryPalette n; }) themeNames);
home = builtins.listToAttrs (map (n: { name = n; value = tryHome n; }) [ "nomarchy" "nixos" ]);
scenarios = builtins.mapAttrs (_: evalToplevel) scenarios;
palettes = builtins.listToAttrs (map (n: { name = n; value = paletteOk n; }) nl.themeNames);
}
NIXEOF
echo "Evaluating opt-in toggle + palette + home matrix (this takes a few minutes)..."
echo "Evaluating combined toggle scenarios + per-palette colour math..."
json="$("${NIX[@]}" eval --impure --json --expr "$EXPR" 2>/tmp/eval-matrix.err)"
status=$?
if [[ $status -ne 0 || -z "$json" ]]; then
@@ -106,13 +136,13 @@ render() {
done < <(echo "$json" | jq -r --arg s "$section" '.[$s] | to_entries[] | "\(.key)\t\(.value)"' | sort)
}
render toggles
render scenarios
render palettes
render home
echo
if [[ $fails -gt 0 ]]; then
echo "$fails scenario(s) failed to evaluate. Re-run the failing one directly for the trace, e.g.:"
echo "$fails scenario(s) failed to evaluate. A combined scenario bundles several"
echo "toggles — re-run the suspects individually for the trace, e.g.:"
echo " nix eval --impure --show-trace --expr 'let f = builtins.getFlake (toString ./.); in (f.nixosConfigurations.default.extendModules { modules = [ { <the toggle> } ]; }).config.system.build.toplevel.drvPath'"
exit 1
fi

2
docs/AGENT.md
View File

@@ -147,7 +147,7 @@ Steps you should follow for any non-trivial change:
```bash
./bin/utils/nomarchy-eval-matrix
```
When you **add or rename an opt-in `nomarchy.*` option**, add a matching scenario to the `toggleScenarios` attrset in that script — otherwise the new surface is unverified and the next eval-time bug in it ships silently (this is exactly how the impermanence assertion and the vscode option rename reached `main`). The matrix also runs in CI (`.gitea/workflows/check.yml`).
When you **add or rename an opt-in `nomarchy.*` option**, fold it into the relevant combined scenario in the `scenarios` attrset in that script (the scenarios bundle many compatible toggles per eval to stay fast on the CI runner) — otherwise the new surface is unverified and the next eval-time bug in it ships silently (this is exactly how the impermanence assertion and the vscode option rename reached `main`). The matrix also runs in CI (`.gitea/workflows/check.yml`).
First clone? Enable the repo's pre-commit hook so `docs/SCRIPTS.md` regenerates whenever you add/modify/remove a `nomarchy-*` script:
```bash
git config core.hooksPath .githooks