feat(stability): default-on memory-pressure protection (earlyoom)
Some checks failed
Check / eval (push) Has been cancelled

modules/nixos/oom.nix — running out of memory now kills the offending
process (with a desktop notification via systembus-notify) instead of
freezing the desktop for minutes until the kernel OOM killer fires.

earlyoom over systemd-oomd, deliberately: oomd kills whole cgroups,
and a Hyprland session runs as ONE scope (nothing here spawns per-app
systemd scopes, unlike GNOME) — under pressure oomd would take out the
entire desktop to save it. earlyoom kills the single largest process
before the thrash point. nixpkgs default-enables oomd in an inert
state (no slices monitored); it is disabled outright so there is one
owner. Session plumbing is --avoid-listed (unanchored — NixOS wrappers
rename comm to .foo-wrapped); no --prefer tuning, largest-RSS
selection already finds the hog. All mkDefault; opt out with
services.earlyoom.enable = false (README note added).

Verified: V0 (flake check) + V2 — new checks.oom-protection
runNixOSTest, executed: a chunked allocator (686 MB peak, 1 GB VM) is
SIGTERM'd by earlyoom in 0.1 s, a bystander unit survives (the
process-level granularity the module exists for), and systemd-oomd is
asserted inactive. 25 s runtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Bernardo Magri
2026-07-04 09:45:21 +01:00
parent 938753273d
commit 5ea4f0c9ac
7 changed files with 133 additions and 17 deletions

View File

@@ -261,11 +261,16 @@ two tables below are split along exactly that line.
| `nomarchy.services.restic.enable` | `false` | Opt-in: scheduled daily restic backup (set `.repository` + `.passwordFile`; 7/4/6 retention; list/restore via the `restic-nomarchy` wrapper) |
Beyond the `nomarchy.*` surface, the system layer turns on the usual
desktop services with `lib.mkDefault` (override natively). One worth
desktop services with `lib.mkDefault` (override natively). Two worth
calling out: **`services.fwupd.enable`** is on by default for firmware
updates via LVFS — it only refreshes metadata, never flashes on its own,
so run `fwupdmgr update` to apply. Disable with `services.fwupd.enable =
false;` on machines without real firmware (VMs/headless).
so run `fwupdmgr update` to apply; disable with `services.fwupd.enable =
false;` on machines without real firmware (VMs/headless). And
**`services.earlyoom`** is on by default so running out of memory kills
the offending process (with a desktop notification) instead of freezing
the desktop — process-level on purpose, since a Hyprland session is one
cgroup and systemd-oomd would kill all of it (oomd is disabled
accordingly). Opt out with `services.earlyoom.enable = false;`.
## 4. How theming works

View File

@@ -35,19 +35,6 @@ likely failure mode; the legacy repo OOM'd on heavy evals). **Stretch
`nix-kvm`, host mode) unlocks the workflow's commented `vm-checks` job
— the VM suite + real builds in CI (upgrades half of item 14 free).
### 3. Memory-pressure protection (no more frozen desktops)
New. A workstation that compiles from source *will* hit memory
exhaustion (the discarded release bump died to a Hyprland OOM; a big
`nix build` can freeze the session today — nothing mitigates OOM).
Ship a default-on userspace OOM killer: pick `systemd-oomd`
(`systemd.oomd` + the NixOS enables for user slices) or `earlyoom`
write the choice + rationale into the commit; consider protecting the
session (Hyprland/greetd) and sacrificing `nix-daemon` build slices
first. **Why:** "rock-stable" means the UI never locks up. **Done
when:** on by default (`lib.mkDefault`), overridable natively; the
choice documented in README's "beyond the surface" note. **Verify:**
V2 — a VM check that memory pressure kills the hog, not the session.
### 4. btrfs-assistant segfault — attempt the nixpkgs override fix
ROADMAP § Snapshot browse/restore. btrfs-assistant 2.2 crashes in
`libbtrfsutil.so.1.4.0` (ABI mismatch, confirmed not a VM artifact); the

View File

@@ -17,6 +17,24 @@ Template:
---
## 2026-07-04 — Memory-pressure protection shipped (loop iteration #3)
- **Task:** BACKLOG NOW#3 — default-on userspace OOM killer.
- **Did:** `modules/nixos/oom.nix`: earlyoom (mkDefault on, desktop
notifications, session-plumbing `--avoid` list, no `--prefer`) +
`systemd.oomd.enable = false`. Choice rationale: Hyprland session is
one cgroup scope → oomd's cgroup-level kill would nuke the desktop;
earlyoom is process-level. Found en route: nixpkgs default-enables
oomd *inert* (enable=true, all slice toggles false). README "beyond
the surface" note added.
- **Verified:** V0 (flake check) + **V2** — new `checks.oom-protection`
runNixOSTest, run for real: hog (chunked allocator, 686 MB peak in a
1 GB VM) SIGTERM'd by earlyoom in 0.1 s, bystander unit survived,
oomd asserted inactive. 25 s test, cheap enough for the suite.
- **Pending:** nothing hardware-specific (behavior fully VM-provable).
- **Next suggestion:** NOW#4 (btrfs-assistant override attempt). Also:
CI run #58 (the .gitea fix) was *running* at commit time — check its
outcome next tick; this push triggers #59.
## 2026-07-04 — CI fix: wrong workflow dir (correction to iteration #2)
- **Task:** Bernardo reported no run despite an active runner + shared
his compose file: the server is **Gitea** (gitea/act_runner), not

View File

@@ -541,6 +541,23 @@ how to override it. Items marked ✓ are shipped.
`cameractrls` for the rare genuine-tuning case; and a follow-up for
portal/Flatpak apps that consume the libcamera path (where the internal IR is
still listed, since libcamera stays on).
- ✓ **Memory-pressure protection (earlyoom, default-on):**
`modules/nixos/oom.nix` — running out of memory kills the offending
process instead of freezing the desktop. **earlyoom over systemd-oomd,
deliberately:** oomd kills whole cgroups, and a Hyprland session is ONE
scope (no per-app systemd scopes here, unlike GNOME) — under pressure it
would take out the entire desktop; earlyoom kills the single largest
process before the thrash point. nixpkgs default-enables oomd *inert*
(no slices monitored) — disabled outright so there's one owner. Session
plumbing is `--avoid`-listed (Hyprland/hyprlock/greetd/waybar/pipewire/
wireplumber/Xwayland/nix-daemon/systemd — unanchored, since NixOS
wrappers rename comm to `.foo-wrapped`); no `--prefer` tuning (largest-
RSS selection already finds the hog); kills raise a desktop notification
(systembus-notify). All `mkDefault` — opt out with
`services.earlyoom.enable = false`. **VM-verified**
(`checks.oom-protection`): a chunked allocator peaked at ~686 MB in a
1 GB VM, earlyoom SIGTERM'd it in 0.1 s, a bystander unit survived, and
oomd is asserted off.
- **Opt-in services & integrations:** the counterpart to the opt-*out*
application suite above — heavier or more personal integrations shipped
**off by default**, each a `nomarchy.services.<name>.enable` toggle a

View File

@@ -207,6 +207,53 @@
'';
};
# Memory-pressure protection (oom.nix): earlyoom must kill a
# runaway allocator BEFORE the kernel thrash point — and must not
# take anything else with it. A bystander unit survives the kill,
# proving the process-level granularity that motivated earlyoom
# over cgroup-level systemd-oomd (which would kill the whole
# Hyprland session scope). Imports only oom.nix so the VM stays
# minimal.
oom-protection = pkgs.testers.runNixOSTest {
name = "nomarchy-oom-protection";
nodes.machine = { ... }: {
imports = [ ./modules/nixos/oom.nix ];
environment.systemPackages = [ pkgs.python3 ];
virtualisation.memorySize = 1024;
};
testScript = ''
machine.wait_for_unit("earlyoom.service")
# One owner: oomd (nixpkgs default-on, inert) is disabled.
machine.fail("systemctl is-active --quiet systemd-oomd.service")
# The session avoid-list made it into the daemon invocation.
machine.succeed("systemctl cat earlyoom.service | grep -q -- --avoid")
# A bystander that must survive (stand-in for the session).
machine.succeed("systemd-run --unit=bystander sleep infinity")
# The hog: allocate in 20 MB chunks with a short sleep, so
# earlyoom's poll wins the race against the kernel OOM killer.
machine.succeed(
"systemd-run --unit=hog python3 -c 'import time; "
"exec(\"a=[]\\nwhile True:\\n a.append(bytearray(20*1024*1024)); time.sleep(0.05)\")'"
)
# earlyoom (not the kernel) pulled the trigger
machine.wait_until_succeeds(
"journalctl -u earlyoom.service | grep -Eiq 'sending SIG(TERM|KILL)'",
timeout=120,
)
# the hog is gone
machine.wait_until_succeeds(
'test "$(systemctl show -p ActiveState --value hog.service)" != "active"',
timeout=60,
)
# and the bystander is untouched.
machine.succeed("systemctl is-active --quiet bystander.service")
'';
};
# The distroId question (roadmap "Distro branding"): set
# distroId = "nomarchy" so /etc/os-release is honest (ID=nomarchy,
# ID_LIKE=nixos). The risk is switch-to-configuration's "is this

View File

@@ -22,7 +22,7 @@ let
'';
in
{
imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ];
imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ./oom.nix ];
config = {
# Distro branding. distroName flows into /etc/os-release PRETTY_NAME,

42
modules/nixos/oom.nix Normal file
View File

@@ -0,0 +1,42 @@
# Memory-pressure protection: keep the desktop alive when RAM runs out.
#
# A workstation that compiles from source WILL exhaust memory eventually —
# a big `nix build`, a runaway eval, a browser tab. The kernel's own OOM
# killer acts only after the system has thrashed itself unresponsive
# (often minutes of frozen desktop) and then picks by badness score,
# which can land on the compositor.
#
# earlyoom over systemd-oomd — a deliberate choice: oomd kills whole
# cgroups, and a Hyprland session runs as ONE scope (nothing spawns
# per-app systemd scopes here, unlike GNOME), so under pressure oomd
# would take out the entire desktop to save it. earlyoom kills a single
# process (highest oom_score ≈ the hog) BEFORE the thrash point — "kill
# the build step, keep the session". nixpkgs default-enables oomd in an
# inert state (no slices monitored); it's disabled outright below so
# there is exactly one owner of the OOM story.
{ lib, ... }:
{
services.earlyoom = {
enable = lib.mkDefault true;
# Desktop toast when something is killed (relayed via
# systembus-notify), so a vanished build/tab is explained rather
# than mysterious.
enableNotifications = lib.mkDefault true;
# Never pick the session plumbing: losing the compositor or the lock
# screen IS the outage this module exists to prevent (and killing a
# Wayland session-lock client trips its go-to-a-tty failsafe). No
# --prefer tuning: highest-memory selection already targets the hog.
# Matched unanchored — NixOS wrappers rename comm to ".foo-wrapped".
extraArgs = lib.mkDefault [
"--avoid"
"(Hyprland|hyprlock|greetd|waybar|pipewire|wireplumber|Xwayland|nix-daemon|systemd)"
];
};
# One owner (see header): oomd ships default-on but inert; make the
# earlyoom choice explicit and total.
systemd.oomd.enable = lib.mkDefault false;
}