From 67400b07dc74b0073ae4ad06d03de3ca664b6aad Mon Sep 17 00:00:00 2001 From: Bernardo Magri Date: Sun, 12 Jul 2026 19:39:46 +0100 Subject: [PATCH] =?UTF-8?q?fix(ci):=20memory-bounded=20eval=20=E2=80=94=20?= =?UTF-8?q?one=20output=20per=20nix=20process?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit check.yml has been permanently red since the runner container was capped at 2 GB (uncapped, it OOM'd the whole 4 GB VPS). Root cause, measured with GNU time (eval-cache off): `nix flake check --no-build` walks every output in ONE evaluator process and peaks at ~6.0 GB RSS on this repo — the checks.* suite alone is ~15 runNixOSTests, each a full NixOS eval, all accumulating in one heap. New tools/ci-eval.sh: identical coverage (all checks.* drvPaths, both nixosConfigurations toplevels, the HM activationPackage — enumerated dynamically, no drift), one fresh nix process per output so the heap is freed between outputs. Cold per-output peaks: worst check 0.99 GB (hardware-toggles), nomarchy toplevel 0.78 GB, HM 0.60 GB; the one outlier is nomarchy-live at 2.69 GB (live-ISO eval), which fits the 2 GB cap only via the container's swap allowance (docker's default --memory-swap is 2x memory). check.yml and bump.yml both call the script now; bump.yml also gains max-jobs=1/cores=2 (it had no limits, so its V1 build gate could spawn parallel builders in the capped container). ROADMAP's stale "no CI today" corrected: bump.yml landed 8fded63 on schedule 2026-07-06. Verified: V1+ — ci-eval.sh green locally end-to-end (3m22s, tree peak 2.75 GB = nomarchy-live's process); per-output peaks measured individually. The decisive proof is the next Actions run on the VPS itself — watch the nomarchy-live step; if it OOMs there, the runner needs swap allowed: --memory=2g --memory-swap=6g. Co-Authored-By: Claude Fable 5 --- .gitea/workflows/bump.yml | 9 +++++++-- .gitea/workflows/check.yml | 12 ++++++++---- agent/JOURNAL.md | 22 ++++++++++++++++++++++ docs/ROADMAP.md | 16 ++++++++++++---- tools/ci-eval.sh | 35 +++++++++++++++++++++++++++++++++++ 5 files changed, 84 insertions(+), 10 deletions(-) create mode 100755 tools/ci-eval.sh diff --git a/.gitea/workflows/bump.yml b/.gitea/workflows/bump.yml index 5448626..7927375 100644 --- a/.gitea/workflows/bump.yml +++ b/.gitea/workflows/bump.yml @@ -36,6 +36,8 @@ jobs: timeout-minutes: 90 env: NIX_CONFIG: | + max-jobs = 1 + cores = 2 experimental-features = nix-command flakes sandbox = false NIX_SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt @@ -68,9 +70,12 @@ jobs: fi id: update - - name: Gate — nix flake check (eval only) + - name: Gate — evaluate every output (eval tier, memory-bounded) + # Per-output processes, not one big `nix flake check --no-build`: + # the single-process walk peaks ~6 GB RSS and OOMs inside the + # runner's 2 GB container cap (details in tools/ci-eval.sh). if: steps.update.outputs.changed == '1' - run: nix flake check --no-build + run: bash tools/ci-eval.sh - name: Gate — V1 build (toplevel + home-manager) if: steps.update.outputs.changed == '1' diff --git a/.gitea/workflows/check.yml b/.gitea/workflows/check.yml index 17a39cb..1e4d8bc 100644 --- a/.gitea/workflows/check.yml +++ b/.gitea/workflows/check.yml @@ -57,11 +57,15 @@ jobs: curl -L "https://releases.nixos.org/nix/nix-${NIX_VERSION}/install" | sh -s -- --no-daemon echo "$HOME/.nix-profile/bin" >> "$GITHUB_PATH" - - name: nix flake check (eval only) - # Full module-system evaluation of every output — both nixos + - name: Evaluate every output (eval tier, memory-bounded) + # Same coverage as `nix flake check --no-build` — both nixos # configs, the home config, the checks.* derivations (instantiated, - # not run) and the downstream template through lib.mkFlake. - run: nix flake check --no-build + # not run) and the downstream template through lib.mkFlake — but + # ONE output per nix process. A single process walking everything + # peaks at ~6.0 GB RSS (measured 2026-07-12): it OOM'd the 4 GB + # VPS itself, and can never fit this runner's 2 GB container cap. + # Per-output processes cap at the largest single output (~1 GB). + run: bash tools/ci-eval.sh - name: Python syntax (all tracked scripts) # Every tracked *.py — theme-sync, the installer composers diff --git a/agent/JOURNAL.md b/agent/JOURNAL.md index 3cf17c2..04ce156 100644 --- a/agent/JOURNAL.md +++ b/agent/JOURNAL.md @@ -19,6 +19,28 @@ Template: --- +## 2026-07-12 — CI OOM: memory-bounded eval for check.yml/bump.yml +- **Task:** Bernardo: check.yml always red since he capped the runner + container at 2 GB (uncapped it OOM'd the whole 4 GB VPS); bump.yml + status unknown. +- **Did:** Measured (GNU time, eval-cache off): one-process `nix flake + check --no-build` peaks **6.0 GB** RSS — that's the whole story. New + `tools/ci-eval.sh`: same coverage, ONE output per nix process (heap + freed between outputs); both workflows now call it. bump.yml also got + max-jobs=1/cores=2. Facts: bump.yml WORKS — landed 8fded63 on + 2026-07-06, on schedule; ROADMAP's "no CI today" was stale → fixed. +- **Verified:** V1+ — ci-eval.sh ran green locally end-to-end (3:22); + per-output cold peaks measured: worst check 0.99 GB + (hardware-toggles), nomarchy toplevel 0.78 GB, HM 0.60 GB, **except + nomarchy-live 2.69 GB** (live ISO eval) — fits only with the + container's swap allowance (docker default --memory-swap = 2×memory); + if Bernardo set swap off, that one output still OOMs → server-side + one-liner (--memory=2g --memory-swap=6g) or a skip-hatch decision. +- **Pending:** real proof is the next push's Actions run on the VPS — + watch nomarchy-live's step; can't be verified from this machine. +- **Next suggestion:** confirm green run on the remote, then hardware + V3 batch. + ## 2026-07-12 — parallel fingerprint-or-password (interactive, promoted) - **Task:** Bernardo promoted the PROPOSED "fingerprint or password in parallel" item live; wants it default-on for fingerprint-PAM users. diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 74b120e..e8f25b4 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -1000,10 +1000,18 @@ Design/decision records and a running log of shipped work (items marked shrinks the security-patch latency the single-input model otherwise imposes (users are gated on the maintainer for nixpkgs CVEs), since a fix is usually already eval/build/VM-tested and one hardware promotion away from `v1` — - worth fast-laning lock-only/security bumps ahead of feature batches. The repo - has **no CI today** (manual `nix flake update` only). Explicitly *not* a - binary cache: compile-from-source is a deliberate values call (Gentoo-style), - so this automates the *config/lock* channel, not artifact distribution. + worth fast-laning lock-only/security bumps ahead of feature batches. + **Shipped since:** `.gitea/workflows/check.yml` (eval tier on every main + push) and `bump.yml` (Monday 05:17 UTC scheduled lock bump, eval + V1 + build gate, auto-lands on green — first landing 8fded63, 2026-07-06). + 2026-07-12: both gates moved from one big `nix flake check --no-build` + (~6.0 GB peak RSS — OOM'd the 4 GB VPS, then could never fit the + runner's 2 GB container cap) to `tools/ci-eval.sh`, one output per nix + process (worst single output: nomarchy-live at 2.69 GB — needs the + container's default swap allowance; everything else ≤ 1 GB). Explicitly + *not* a binary cache: compile-from-source is a deliberate values call + (Gentoo-style), so this automates the *config/lock* channel, not + artifact distribution. - ✓ **Opt-in auto-commit of state mutations (in-flake state, Phase 4):** `settings.autoCommit` (menu: **System › Auto-commit**, self-gated on the flake being a git repo) makes every `apply`/`set` also `git commit` diff --git a/tools/ci-eval.sh b/tools/ci-eval.sh new file mode 100755 index 0000000..251f530 --- /dev/null +++ b/tools/ci-eval.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +# CI eval tier — same coverage as `nix flake check --no-build`, bounded +# memory. One evaluator process walking every output accumulates ~6.0 GB +# peak RSS on this repo (measured 2026-07-12, eval-cache off): that +# OOM'd the whole 4 GB VPS behind git.bemagri.xyz, and can never fit the +# runner's 2 GB container cap. Evaluating ONE output per fresh nix +# process caps the peak at the largest single output (~0.93 GB, +# downstream-template-system): the heap is freed when each process +# exits. Wall time is worse (shared nixpkgs re-eval per process); +# irrelevant next to the memory cap. Used by .gitea/workflows/check.yml +# and bump.yml; runs anywhere (`bash tools/ci-eval.sh`). +set -euo pipefail + +sys=${1:-x86_64-linux} + +list() { + nix eval --raw ".#$1" \ + --apply 'a: builtins.concatStringsSep "\n" (builtins.attrNames a)' +} + +evaluate() { + echo "eval: $1" + nix eval --raw ".#$1" >/dev/null +} + +for a in $(list "checks.$sys"); do + evaluate "checks.$sys.$a.drvPath" +done +for c in $(list nixosConfigurations); do + evaluate "nixosConfigurations.$c.config.system.build.toplevel.drvPath" +done +for h in $(list homeConfigurations); do + evaluate "homeConfigurations.$h.activationPackage.drvPath" +done +echo "ci-eval: all outputs evaluated."