fix(ci): memory-bounded eval — one output per nix process
Some checks failed
Check / eval (push) Failing after 7m59s
Some checks failed
Check / eval (push) Failing after 7m59s
check.yml has been permanently red since the runner container was
capped at 2 GB (uncapped, it OOM'd the whole 4 GB VPS). Root cause,
measured with GNU time (eval-cache off): `nix flake check --no-build`
walks every output in ONE evaluator process and peaks at ~6.0 GB RSS
on this repo — the checks.* suite alone is ~15 runNixOSTests, each a
full NixOS eval, all accumulating in one heap.
New tools/ci-eval.sh: identical coverage (all checks.* drvPaths, both
nixosConfigurations toplevels, the HM activationPackage — enumerated
dynamically, no drift), one fresh nix process per output so the heap is
freed between outputs. Cold per-output peaks: worst check 0.99 GB
(hardware-toggles), nomarchy toplevel 0.78 GB, HM 0.60 GB; the one
outlier is nomarchy-live at 2.69 GB (live-ISO eval), which fits the
2 GB cap only via the container's swap allowance (docker's default
--memory-swap is 2x memory). check.yml and bump.yml both call the
script now; bump.yml also gains max-jobs=1/cores=2 (it had no limits,
so its V1 build gate could spawn parallel builders in the capped
container). ROADMAP's stale "no CI today" corrected: bump.yml landed
8fded63 on schedule 2026-07-06.
Verified: V1+ — ci-eval.sh green locally end-to-end (3m22s, tree peak
2.75 GB = nomarchy-live's process); per-output peaks measured
individually. The decisive proof is the next Actions run on the VPS
itself — watch the nomarchy-live step; if it OOMs there, the runner
needs swap allowed: --memory=2g --memory-swap=6g.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -36,6 +36,8 @@ jobs:
|
|||||||
timeout-minutes: 90
|
timeout-minutes: 90
|
||||||
env:
|
env:
|
||||||
NIX_CONFIG: |
|
NIX_CONFIG: |
|
||||||
|
max-jobs = 1
|
||||||
|
cores = 2
|
||||||
experimental-features = nix-command flakes
|
experimental-features = nix-command flakes
|
||||||
sandbox = false
|
sandbox = false
|
||||||
NIX_SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
|
NIX_SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt
|
||||||
@@ -68,9 +70,12 @@ jobs:
|
|||||||
fi
|
fi
|
||||||
id: update
|
id: update
|
||||||
|
|
||||||
- name: Gate — nix flake check (eval only)
|
- name: Gate — evaluate every output (eval tier, memory-bounded)
|
||||||
|
# Per-output processes, not one big `nix flake check --no-build`:
|
||||||
|
# the single-process walk peaks ~6 GB RSS and OOMs inside the
|
||||||
|
# runner's 2 GB container cap (details in tools/ci-eval.sh).
|
||||||
if: steps.update.outputs.changed == '1'
|
if: steps.update.outputs.changed == '1'
|
||||||
run: nix flake check --no-build
|
run: bash tools/ci-eval.sh
|
||||||
|
|
||||||
- name: Gate — V1 build (toplevel + home-manager)
|
- name: Gate — V1 build (toplevel + home-manager)
|
||||||
if: steps.update.outputs.changed == '1'
|
if: steps.update.outputs.changed == '1'
|
||||||
|
|||||||
@@ -57,11 +57,15 @@ jobs:
|
|||||||
curl -L "https://releases.nixos.org/nix/nix-${NIX_VERSION}/install" | sh -s -- --no-daemon
|
curl -L "https://releases.nixos.org/nix/nix-${NIX_VERSION}/install" | sh -s -- --no-daemon
|
||||||
echo "$HOME/.nix-profile/bin" >> "$GITHUB_PATH"
|
echo "$HOME/.nix-profile/bin" >> "$GITHUB_PATH"
|
||||||
|
|
||||||
- name: nix flake check (eval only)
|
- name: Evaluate every output (eval tier, memory-bounded)
|
||||||
# Full module-system evaluation of every output — both nixos
|
# Same coverage as `nix flake check --no-build` — both nixos
|
||||||
# configs, the home config, the checks.* derivations (instantiated,
|
# configs, the home config, the checks.* derivations (instantiated,
|
||||||
# not run) and the downstream template through lib.mkFlake.
|
# not run) and the downstream template through lib.mkFlake — but
|
||||||
run: nix flake check --no-build
|
# ONE output per nix process. A single process walking everything
|
||||||
|
# peaks at ~6.0 GB RSS (measured 2026-07-12): it OOM'd the 4 GB
|
||||||
|
# VPS itself, and can never fit this runner's 2 GB container cap.
|
||||||
|
# Per-output processes cap at the largest single output (~1 GB).
|
||||||
|
run: bash tools/ci-eval.sh
|
||||||
|
|
||||||
- name: Python syntax (all tracked scripts)
|
- name: Python syntax (all tracked scripts)
|
||||||
# Every tracked *.py — theme-sync, the installer composers
|
# Every tracked *.py — theme-sync, the installer composers
|
||||||
|
|||||||
@@ -19,6 +19,28 @@ Template:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-07-12 — CI OOM: memory-bounded eval for check.yml/bump.yml
|
||||||
|
- **Task:** Bernardo: check.yml always red since he capped the runner
|
||||||
|
container at 2 GB (uncapped it OOM'd the whole 4 GB VPS); bump.yml
|
||||||
|
status unknown.
|
||||||
|
- **Did:** Measured (GNU time, eval-cache off): one-process `nix flake
|
||||||
|
check --no-build` peaks **6.0 GB** RSS — that's the whole story. New
|
||||||
|
`tools/ci-eval.sh`: same coverage, ONE output per nix process (heap
|
||||||
|
freed between outputs); both workflows now call it. bump.yml also got
|
||||||
|
max-jobs=1/cores=2. Facts: bump.yml WORKS — landed 8fded63 on
|
||||||
|
2026-07-06, on schedule; ROADMAP's "no CI today" was stale → fixed.
|
||||||
|
- **Verified:** V1+ — ci-eval.sh ran green locally end-to-end (3:22);
|
||||||
|
per-output cold peaks measured: worst check 0.99 GB
|
||||||
|
(hardware-toggles), nomarchy toplevel 0.78 GB, HM 0.60 GB, **except
|
||||||
|
nomarchy-live 2.69 GB** (live ISO eval) — fits only with the
|
||||||
|
container's swap allowance (docker default --memory-swap = 2×memory);
|
||||||
|
if Bernardo set swap off, that one output still OOMs → server-side
|
||||||
|
one-liner (--memory=2g --memory-swap=6g) or a skip-hatch decision.
|
||||||
|
- **Pending:** real proof is the next push's Actions run on the VPS —
|
||||||
|
watch nomarchy-live's step; can't be verified from this machine.
|
||||||
|
- **Next suggestion:** confirm green run on the remote, then hardware
|
||||||
|
V3 batch.
|
||||||
|
|
||||||
## 2026-07-12 — parallel fingerprint-or-password (interactive, promoted)
|
## 2026-07-12 — parallel fingerprint-or-password (interactive, promoted)
|
||||||
- **Task:** Bernardo promoted the PROPOSED "fingerprint or password in
|
- **Task:** Bernardo promoted the PROPOSED "fingerprint or password in
|
||||||
parallel" item live; wants it default-on for fingerprint-PAM users.
|
parallel" item live; wants it default-on for fingerprint-PAM users.
|
||||||
|
|||||||
@@ -1000,10 +1000,18 @@ Design/decision records and a running log of shipped work (items marked
|
|||||||
shrinks the security-patch latency the single-input model otherwise imposes
|
shrinks the security-patch latency the single-input model otherwise imposes
|
||||||
(users are gated on the maintainer for nixpkgs CVEs), since a fix is usually
|
(users are gated on the maintainer for nixpkgs CVEs), since a fix is usually
|
||||||
already eval/build/VM-tested and one hardware promotion away from `v1` —
|
already eval/build/VM-tested and one hardware promotion away from `v1` —
|
||||||
worth fast-laning lock-only/security bumps ahead of feature batches. The repo
|
worth fast-laning lock-only/security bumps ahead of feature batches.
|
||||||
has **no CI today** (manual `nix flake update` only). Explicitly *not* a
|
**Shipped since:** `.gitea/workflows/check.yml` (eval tier on every main
|
||||||
binary cache: compile-from-source is a deliberate values call (Gentoo-style),
|
push) and `bump.yml` (Monday 05:17 UTC scheduled lock bump, eval + V1
|
||||||
so this automates the *config/lock* channel, not artifact distribution.
|
build gate, auto-lands on green — first landing 8fded63, 2026-07-06).
|
||||||
|
2026-07-12: both gates moved from one big `nix flake check --no-build`
|
||||||
|
(~6.0 GB peak RSS — OOM'd the 4 GB VPS, then could never fit the
|
||||||
|
runner's 2 GB container cap) to `tools/ci-eval.sh`, one output per nix
|
||||||
|
process (worst single output: nomarchy-live at 2.69 GB — needs the
|
||||||
|
container's default swap allowance; everything else ≤ 1 GB). Explicitly
|
||||||
|
*not* a binary cache: compile-from-source is a deliberate values call
|
||||||
|
(Gentoo-style), so this automates the *config/lock* channel, not
|
||||||
|
artifact distribution.
|
||||||
- ✓ **Opt-in auto-commit of state mutations (in-flake state, Phase 4):**
|
- ✓ **Opt-in auto-commit of state mutations (in-flake state, Phase 4):**
|
||||||
`settings.autoCommit` (menu: **System › Auto-commit**, self-gated on the
|
`settings.autoCommit` (menu: **System › Auto-commit**, self-gated on the
|
||||||
flake being a git repo) makes every `apply`/`set` also `git commit`
|
flake being a git repo) makes every `apply`/`set` also `git commit`
|
||||||
|
|||||||
35
tools/ci-eval.sh
Executable file
35
tools/ci-eval.sh
Executable file
@@ -0,0 +1,35 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# CI eval tier — same coverage as `nix flake check --no-build`, bounded
|
||||||
|
# memory. One evaluator process walking every output accumulates ~6.0 GB
|
||||||
|
# peak RSS on this repo (measured 2026-07-12, eval-cache off): that
|
||||||
|
# OOM'd the whole 4 GB VPS behind git.bemagri.xyz, and can never fit the
|
||||||
|
# runner's 2 GB container cap. Evaluating ONE output per fresh nix
|
||||||
|
# process caps the peak at the largest single output (~0.93 GB,
|
||||||
|
# downstream-template-system): the heap is freed when each process
|
||||||
|
# exits. Wall time is worse (shared nixpkgs re-eval per process);
|
||||||
|
# irrelevant next to the memory cap. Used by .gitea/workflows/check.yml
|
||||||
|
# and bump.yml; runs anywhere (`bash tools/ci-eval.sh`).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
sys=${1:-x86_64-linux}
|
||||||
|
|
||||||
|
list() {
|
||||||
|
nix eval --raw ".#$1" \
|
||||||
|
--apply 'a: builtins.concatStringsSep "\n" (builtins.attrNames a)'
|
||||||
|
}
|
||||||
|
|
||||||
|
evaluate() {
|
||||||
|
echo "eval: $1"
|
||||||
|
nix eval --raw ".#$1" >/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
for a in $(list "checks.$sys"); do
|
||||||
|
evaluate "checks.$sys.$a.drvPath"
|
||||||
|
done
|
||||||
|
for c in $(list nixosConfigurations); do
|
||||||
|
evaluate "nixosConfigurations.$c.config.system.build.toplevel.drvPath"
|
||||||
|
done
|
||||||
|
for h in $(list homeConfigurations); do
|
||||||
|
evaluate "homeConfigurations.$h.activationPackage.drvPath"
|
||||||
|
done
|
||||||
|
echo "ci-eval: all outputs evaluated."
|
||||||
Reference in New Issue
Block a user