From 5ea4f0c9ac4ad87a82568d7cf7cffd040ce9bc1e Mon Sep 17 00:00:00 2001 From: Bernardo Magri Date: Sat, 4 Jul 2026 09:45:21 +0100 Subject: [PATCH] feat(stability): default-on memory-pressure protection (earlyoom) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit modules/nixos/oom.nix — running out of memory now kills the offending process (with a desktop notification via systembus-notify) instead of freezing the desktop for minutes until the kernel OOM killer fires. earlyoom over systemd-oomd, deliberately: oomd kills whole cgroups, and a Hyprland session runs as ONE scope (nothing here spawns per-app systemd scopes, unlike GNOME) — under pressure oomd would take out the entire desktop to save it. earlyoom kills the single largest process before the thrash point. nixpkgs default-enables oomd in an inert state (no slices monitored); it is disabled outright so there is one owner. Session plumbing is --avoid-listed (unanchored — NixOS wrappers rename comm to .foo-wrapped); no --prefer tuning, largest-RSS selection already finds the hog. All mkDefault; opt out with services.earlyoom.enable = false (README note added). Verified: V0 (flake check) + V2 — new checks.oom-protection runNixOSTest, executed: a chunked allocator (686 MB peak, 1 GB VM) is SIGTERM'd by earlyoom in 0.1 s, a bystander unit survives (the process-level granularity the module exists for), and systemd-oomd is asserted inactive. 25 s runtime. Co-Authored-By: Claude Fable 5 --- README.md | 11 ++++++--- agent/BACKLOG.md | 13 ----------- agent/JOURNAL.md | 18 +++++++++++++++ docs/ROADMAP.md | 17 ++++++++++++++ flake.nix | 47 +++++++++++++++++++++++++++++++++++++++ modules/nixos/default.nix | 2 +- modules/nixos/oom.nix | 42 ++++++++++++++++++++++++++++++++++ 7 files changed, 133 insertions(+), 17 deletions(-) create mode 100644 modules/nixos/oom.nix diff --git a/README.md b/README.md index 89f7a08..dd3b5a6 100644 --- a/README.md +++ b/README.md @@ -261,11 +261,16 @@ two tables below are split along exactly that line. | `nomarchy.services.restic.enable` | `false` | Opt-in: scheduled daily restic backup (set `.repository` + `.passwordFile`; 7/4/6 retention; list/restore via the `restic-nomarchy` wrapper) | Beyond the `nomarchy.*` surface, the system layer turns on the usual -desktop services with `lib.mkDefault` (override natively). One worth +desktop services with `lib.mkDefault` (override natively). Two worth calling out: **`services.fwupd.enable`** is on by default for firmware updates via LVFS — it only refreshes metadata, never flashes on its own, -so run `fwupdmgr update` to apply. Disable with `services.fwupd.enable = -false;` on machines without real firmware (VMs/headless). +so run `fwupdmgr update` to apply; disable with `services.fwupd.enable = +false;` on machines without real firmware (VMs/headless). And +**`services.earlyoom`** is on by default so running out of memory kills +the offending process (with a desktop notification) instead of freezing +the desktop — process-level on purpose, since a Hyprland session is one +cgroup and systemd-oomd would kill all of it (oomd is disabled +accordingly). Opt out with `services.earlyoom.enable = false;`. ## 4. How theming works diff --git a/agent/BACKLOG.md b/agent/BACKLOG.md index 22d7bc3..35952d6 100644 --- a/agent/BACKLOG.md +++ b/agent/BACKLOG.md @@ -35,19 +35,6 @@ likely failure mode; the legacy repo OOM'd on heavy evals). **Stretch `nix-kvm`, host mode) unlocks the workflow's commented `vm-checks` job — the VM suite + real builds in CI (upgrades half of item 14 free). -### 3. Memory-pressure protection (no more frozen desktops) -New. A workstation that compiles from source *will* hit memory -exhaustion (the discarded release bump died to a Hyprland OOM; a big -`nix build` can freeze the session today — nothing mitigates OOM). -Ship a default-on userspace OOM killer: pick `systemd-oomd` -(`systemd.oomd` + the NixOS enables for user slices) or `earlyoom` — -write the choice + rationale into the commit; consider protecting the -session (Hyprland/greetd) and sacrificing `nix-daemon` build slices -first. **Why:** "rock-stable" means the UI never locks up. **Done -when:** on by default (`lib.mkDefault`), overridable natively; the -choice documented in README's "beyond the surface" note. **Verify:** -V2 — a VM check that memory pressure kills the hog, not the session. - ### 4. btrfs-assistant segfault — attempt the nixpkgs override fix ROADMAP § Snapshot browse/restore. btrfs-assistant 2.2 crashes in `libbtrfsutil.so.1.4.0` (ABI mismatch, confirmed not a VM artifact); the diff --git a/agent/JOURNAL.md b/agent/JOURNAL.md index e16204b..b05e094 100644 --- a/agent/JOURNAL.md +++ b/agent/JOURNAL.md @@ -17,6 +17,24 @@ Template: --- +## 2026-07-04 — Memory-pressure protection shipped (loop iteration #3) +- **Task:** BACKLOG NOW#3 — default-on userspace OOM killer. +- **Did:** `modules/nixos/oom.nix`: earlyoom (mkDefault on, desktop + notifications, session-plumbing `--avoid` list, no `--prefer`) + + `systemd.oomd.enable = false`. Choice rationale: Hyprland session is + one cgroup scope → oomd's cgroup-level kill would nuke the desktop; + earlyoom is process-level. Found en route: nixpkgs default-enables + oomd *inert* (enable=true, all slice toggles false). README "beyond + the surface" note added. +- **Verified:** V0 (flake check) + **V2** — new `checks.oom-protection` + runNixOSTest, run for real: hog (chunked allocator, 686 MB peak in a + 1 GB VM) SIGTERM'd by earlyoom in 0.1 s, bystander unit survived, + oomd asserted inactive. 25 s test, cheap enough for the suite. +- **Pending:** nothing hardware-specific (behavior fully VM-provable). +- **Next suggestion:** NOW#4 (btrfs-assistant override attempt). Also: + CI run #58 (the .gitea fix) was *running* at commit time — check its + outcome next tick; this push triggers #59. + ## 2026-07-04 — CI fix: wrong workflow dir (correction to iteration #2) - **Task:** Bernardo reported no run despite an active runner + shared his compose file: the server is **Gitea** (gitea/act_runner), not diff --git a/docs/ROADMAP.md b/docs/ROADMAP.md index 105bb97..2d34a9e 100644 --- a/docs/ROADMAP.md +++ b/docs/ROADMAP.md @@ -541,6 +541,23 @@ how to override it. Items marked ✓ are shipped. `cameractrls` for the rare genuine-tuning case; and a follow-up for portal/Flatpak apps that consume the libcamera path (where the internal IR is still listed, since libcamera stays on). +- ✓ **Memory-pressure protection (earlyoom, default-on):** + `modules/nixos/oom.nix` — running out of memory kills the offending + process instead of freezing the desktop. **earlyoom over systemd-oomd, + deliberately:** oomd kills whole cgroups, and a Hyprland session is ONE + scope (no per-app systemd scopes here, unlike GNOME) — under pressure it + would take out the entire desktop; earlyoom kills the single largest + process before the thrash point. nixpkgs default-enables oomd *inert* + (no slices monitored) — disabled outright so there's one owner. Session + plumbing is `--avoid`-listed (Hyprland/hyprlock/greetd/waybar/pipewire/ + wireplumber/Xwayland/nix-daemon/systemd — unanchored, since NixOS + wrappers rename comm to `.foo-wrapped`); no `--prefer` tuning (largest- + RSS selection already finds the hog); kills raise a desktop notification + (systembus-notify). All `mkDefault` — opt out with + `services.earlyoom.enable = false`. **VM-verified** + (`checks.oom-protection`): a chunked allocator peaked at ~686 MB in a + 1 GB VM, earlyoom SIGTERM'd it in 0.1 s, a bystander unit survived, and + oomd is asserted off. - **Opt-in services & integrations:** the counterpart to the opt-*out* application suite above — heavier or more personal integrations shipped **off by default**, each a `nomarchy.services..enable` toggle a diff --git a/flake.nix b/flake.nix index e56c2e0..b8636c9 100644 --- a/flake.nix +++ b/flake.nix @@ -207,6 +207,53 @@ ''; }; + # Memory-pressure protection (oom.nix): earlyoom must kill a + # runaway allocator BEFORE the kernel thrash point — and must not + # take anything else with it. A bystander unit survives the kill, + # proving the process-level granularity that motivated earlyoom + # over cgroup-level systemd-oomd (which would kill the whole + # Hyprland session scope). Imports only oom.nix so the VM stays + # minimal. + oom-protection = pkgs.testers.runNixOSTest { + name = "nomarchy-oom-protection"; + nodes.machine = { ... }: { + imports = [ ./modules/nixos/oom.nix ]; + environment.systemPackages = [ pkgs.python3 ]; + virtualisation.memorySize = 1024; + }; + testScript = '' + machine.wait_for_unit("earlyoom.service") + + # One owner: oomd (nixpkgs default-on, inert) is disabled. + machine.fail("systemctl is-active --quiet systemd-oomd.service") + # The session avoid-list made it into the daemon invocation. + machine.succeed("systemctl cat earlyoom.service | grep -q -- --avoid") + + # A bystander that must survive (stand-in for the session). + machine.succeed("systemd-run --unit=bystander sleep infinity") + + # The hog: allocate in 20 MB chunks with a short sleep, so + # earlyoom's poll wins the race against the kernel OOM killer. + machine.succeed( + "systemd-run --unit=hog python3 -c 'import time; " + "exec(\"a=[]\\nwhile True:\\n a.append(bytearray(20*1024*1024)); time.sleep(0.05)\")'" + ) + + # earlyoom (not the kernel) pulled the trigger… + machine.wait_until_succeeds( + "journalctl -u earlyoom.service | grep -Eiq 'sending SIG(TERM|KILL)'", + timeout=120, + ) + # …the hog is gone… + machine.wait_until_succeeds( + 'test "$(systemctl show -p ActiveState --value hog.service)" != "active"', + timeout=60, + ) + # …and the bystander is untouched. + machine.succeed("systemctl is-active --quiet bystander.service") + ''; + }; + # The distroId question (roadmap "Distro branding"): set # distroId = "nomarchy" so /etc/os-release is honest (ID=nomarchy, # ID_LIKE=nixos). The risk is switch-to-configuration's "is this diff --git a/modules/nixos/default.nix b/modules/nixos/default.nix index 70d04da..5d42070 100644 --- a/modules/nixos/default.nix +++ b/modules/nixos/default.nix @@ -22,7 +22,7 @@ let ''; in { - imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ]; + imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ./oom.nix ]; config = { # Distro branding. distroName flows into /etc/os-release PRETTY_NAME, diff --git a/modules/nixos/oom.nix b/modules/nixos/oom.nix new file mode 100644 index 0000000..13f378e --- /dev/null +++ b/modules/nixos/oom.nix @@ -0,0 +1,42 @@ +# Memory-pressure protection: keep the desktop alive when RAM runs out. +# +# A workstation that compiles from source WILL exhaust memory eventually — +# a big `nix build`, a runaway eval, a browser tab. The kernel's own OOM +# killer acts only after the system has thrashed itself unresponsive +# (often minutes of frozen desktop) and then picks by badness score, +# which can land on the compositor. +# +# earlyoom over systemd-oomd — a deliberate choice: oomd kills whole +# cgroups, and a Hyprland session runs as ONE scope (nothing spawns +# per-app systemd scopes here, unlike GNOME), so under pressure oomd +# would take out the entire desktop to save it. earlyoom kills a single +# process (highest oom_score ≈ the hog) BEFORE the thrash point — "kill +# the build step, keep the session". nixpkgs default-enables oomd in an +# inert state (no slices monitored); it's disabled outright below so +# there is exactly one owner of the OOM story. +{ lib, ... }: + +{ + services.earlyoom = { + enable = lib.mkDefault true; + + # Desktop toast when something is killed (relayed via + # systembus-notify), so a vanished build/tab is explained rather + # than mysterious. + enableNotifications = lib.mkDefault true; + + # Never pick the session plumbing: losing the compositor or the lock + # screen IS the outage this module exists to prevent (and killing a + # Wayland session-lock client trips its go-to-a-tty failsafe). No + # --prefer tuning: highest-memory selection already targets the hog. + # Matched unanchored — NixOS wrappers rename comm to ".foo-wrapped". + extraArgs = lib.mkDefault [ + "--avoid" + "(Hyprland|hyprlock|greetd|waybar|pipewire|wireplumber|Xwayland|nix-daemon|systemd)" + ]; + }; + + # One owner (see header): oomd ships default-on but inert; make the + # earlyoom choice explicit and total. + systemd.oomd.enable = lib.mkDefault false; +}