feat(stability): default-on memory-pressure protection (earlyoom)
Some checks failed
Check / eval (push) Has been cancelled
Some checks failed
Check / eval (push) Has been cancelled
modules/nixos/oom.nix — running out of memory now kills the offending process (with a desktop notification via systembus-notify) instead of freezing the desktop for minutes until the kernel OOM killer fires. earlyoom over systemd-oomd, deliberately: oomd kills whole cgroups, and a Hyprland session runs as ONE scope (nothing here spawns per-app systemd scopes, unlike GNOME) — under pressure oomd would take out the entire desktop to save it. earlyoom kills the single largest process before the thrash point. nixpkgs default-enables oomd in an inert state (no slices monitored); it is disabled outright so there is one owner. Session plumbing is --avoid-listed (unanchored — NixOS wrappers rename comm to .foo-wrapped); no --prefer tuning, largest-RSS selection already finds the hog. All mkDefault; opt out with services.earlyoom.enable = false (README note added). Verified: V0 (flake check) + V2 — new checks.oom-protection runNixOSTest, executed: a chunked allocator (686 MB peak, 1 GB VM) is SIGTERM'd by earlyoom in 0.1 s, a bystander unit survives (the process-level granularity the module exists for), and systemd-oomd is asserted inactive. 25 s runtime. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
11
README.md
11
README.md
@@ -261,11 +261,16 @@ two tables below are split along exactly that line.
|
||||
| `nomarchy.services.restic.enable` | `false` | Opt-in: scheduled daily restic backup (set `.repository` + `.passwordFile`; 7/4/6 retention; list/restore via the `restic-nomarchy` wrapper) |
|
||||
|
||||
Beyond the `nomarchy.*` surface, the system layer turns on the usual
|
||||
desktop services with `lib.mkDefault` (override natively). One worth
|
||||
desktop services with `lib.mkDefault` (override natively). Two worth
|
||||
calling out: **`services.fwupd.enable`** is on by default for firmware
|
||||
updates via LVFS — it only refreshes metadata, never flashes on its own,
|
||||
so run `fwupdmgr update` to apply. Disable with `services.fwupd.enable =
|
||||
false;` on machines without real firmware (VMs/headless).
|
||||
so run `fwupdmgr update` to apply; disable with `services.fwupd.enable =
|
||||
false;` on machines without real firmware (VMs/headless). And
|
||||
**`services.earlyoom`** is on by default so running out of memory kills
|
||||
the offending process (with a desktop notification) instead of freezing
|
||||
the desktop — process-level on purpose, since a Hyprland session is one
|
||||
cgroup and systemd-oomd would kill all of it (oomd is disabled
|
||||
accordingly). Opt out with `services.earlyoom.enable = false;`.
|
||||
|
||||
## 4. How theming works
|
||||
|
||||
|
||||
@@ -35,19 +35,6 @@ likely failure mode; the legacy repo OOM'd on heavy evals). **Stretch
|
||||
`nix-kvm`, host mode) unlocks the workflow's commented `vm-checks` job
|
||||
— the VM suite + real builds in CI (upgrades half of item 14 free).
|
||||
|
||||
### 3. Memory-pressure protection (no more frozen desktops)
|
||||
New. A workstation that compiles from source *will* hit memory
|
||||
exhaustion (the discarded release bump died to a Hyprland OOM; a big
|
||||
`nix build` can freeze the session today — nothing mitigates OOM).
|
||||
Ship a default-on userspace OOM killer: pick `systemd-oomd`
|
||||
(`systemd.oomd` + the NixOS enables for user slices) or `earlyoom` —
|
||||
write the choice + rationale into the commit; consider protecting the
|
||||
session (Hyprland/greetd) and sacrificing `nix-daemon` build slices
|
||||
first. **Why:** "rock-stable" means the UI never locks up. **Done
|
||||
when:** on by default (`lib.mkDefault`), overridable natively; the
|
||||
choice documented in README's "beyond the surface" note. **Verify:**
|
||||
V2 — a VM check that memory pressure kills the hog, not the session.
|
||||
|
||||
### 4. btrfs-assistant segfault — attempt the nixpkgs override fix
|
||||
ROADMAP § Snapshot browse/restore. btrfs-assistant 2.2 crashes in
|
||||
`libbtrfsutil.so.1.4.0` (ABI mismatch, confirmed not a VM artifact); the
|
||||
|
||||
@@ -17,6 +17,24 @@ Template:
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-04 — Memory-pressure protection shipped (loop iteration #3)
|
||||
- **Task:** BACKLOG NOW#3 — default-on userspace OOM killer.
|
||||
- **Did:** `modules/nixos/oom.nix`: earlyoom (mkDefault on, desktop
|
||||
notifications, session-plumbing `--avoid` list, no `--prefer`) +
|
||||
`systemd.oomd.enable = false`. Choice rationale: Hyprland session is
|
||||
one cgroup scope → oomd's cgroup-level kill would nuke the desktop;
|
||||
earlyoom is process-level. Found en route: nixpkgs default-enables
|
||||
oomd *inert* (enable=true, all slice toggles false). README "beyond
|
||||
the surface" note added.
|
||||
- **Verified:** V0 (flake check) + **V2** — new `checks.oom-protection`
|
||||
runNixOSTest, run for real: hog (chunked allocator, 686 MB peak in a
|
||||
1 GB VM) SIGTERM'd by earlyoom in 0.1 s, bystander unit survived,
|
||||
oomd asserted inactive. 25 s test, cheap enough for the suite.
|
||||
- **Pending:** nothing hardware-specific (behavior fully VM-provable).
|
||||
- **Next suggestion:** NOW#4 (btrfs-assistant override attempt). Also:
|
||||
CI run #58 (the .gitea fix) was *running* at commit time — check its
|
||||
outcome next tick; this push triggers #59.
|
||||
|
||||
## 2026-07-04 — CI fix: wrong workflow dir (correction to iteration #2)
|
||||
- **Task:** Bernardo reported no run despite an active runner + shared
|
||||
his compose file: the server is **Gitea** (gitea/act_runner), not
|
||||
|
||||
@@ -541,6 +541,23 @@ how to override it. Items marked ✓ are shipped.
|
||||
`cameractrls` for the rare genuine-tuning case; and a follow-up for
|
||||
portal/Flatpak apps that consume the libcamera path (where the internal IR is
|
||||
still listed, since libcamera stays on).
|
||||
- ✓ **Memory-pressure protection (earlyoom, default-on):**
|
||||
`modules/nixos/oom.nix` — running out of memory kills the offending
|
||||
process instead of freezing the desktop. **earlyoom over systemd-oomd,
|
||||
deliberately:** oomd kills whole cgroups, and a Hyprland session is ONE
|
||||
scope (no per-app systemd scopes here, unlike GNOME) — under pressure it
|
||||
would take out the entire desktop; earlyoom kills the single largest
|
||||
process before the thrash point. nixpkgs default-enables oomd *inert*
|
||||
(no slices monitored) — disabled outright so there's one owner. Session
|
||||
plumbing is `--avoid`-listed (Hyprland/hyprlock/greetd/waybar/pipewire/
|
||||
wireplumber/Xwayland/nix-daemon/systemd — unanchored, since NixOS
|
||||
wrappers rename comm to `.foo-wrapped`); no `--prefer` tuning (largest-
|
||||
RSS selection already finds the hog); kills raise a desktop notification
|
||||
(systembus-notify). All `mkDefault` — opt out with
|
||||
`services.earlyoom.enable = false`. **VM-verified**
|
||||
(`checks.oom-protection`): a chunked allocator peaked at ~686 MB in a
|
||||
1 GB VM, earlyoom SIGTERM'd it in 0.1 s, a bystander unit survived, and
|
||||
oomd is asserted off.
|
||||
- **Opt-in services & integrations:** the counterpart to the opt-*out*
|
||||
application suite above — heavier or more personal integrations shipped
|
||||
**off by default**, each a `nomarchy.services.<name>.enable` toggle a
|
||||
|
||||
47
flake.nix
47
flake.nix
@@ -207,6 +207,53 @@
|
||||
'';
|
||||
};
|
||||
|
||||
# Memory-pressure protection (oom.nix): earlyoom must kill a
|
||||
# runaway allocator BEFORE the kernel thrash point — and must not
|
||||
# take anything else with it. A bystander unit survives the kill,
|
||||
# proving the process-level granularity that motivated earlyoom
|
||||
# over cgroup-level systemd-oomd (which would kill the whole
|
||||
# Hyprland session scope). Imports only oom.nix so the VM stays
|
||||
# minimal.
|
||||
oom-protection = pkgs.testers.runNixOSTest {
|
||||
name = "nomarchy-oom-protection";
|
||||
nodes.machine = { ... }: {
|
||||
imports = [ ./modules/nixos/oom.nix ];
|
||||
environment.systemPackages = [ pkgs.python3 ];
|
||||
virtualisation.memorySize = 1024;
|
||||
};
|
||||
testScript = ''
|
||||
machine.wait_for_unit("earlyoom.service")
|
||||
|
||||
# One owner: oomd (nixpkgs default-on, inert) is disabled.
|
||||
machine.fail("systemctl is-active --quiet systemd-oomd.service")
|
||||
# The session avoid-list made it into the daemon invocation.
|
||||
machine.succeed("systemctl cat earlyoom.service | grep -q -- --avoid")
|
||||
|
||||
# A bystander that must survive (stand-in for the session).
|
||||
machine.succeed("systemd-run --unit=bystander sleep infinity")
|
||||
|
||||
# The hog: allocate in 20 MB chunks with a short sleep, so
|
||||
# earlyoom's poll wins the race against the kernel OOM killer.
|
||||
machine.succeed(
|
||||
"systemd-run --unit=hog python3 -c 'import time; "
|
||||
"exec(\"a=[]\\nwhile True:\\n a.append(bytearray(20*1024*1024)); time.sleep(0.05)\")'"
|
||||
)
|
||||
|
||||
# earlyoom (not the kernel) pulled the trigger…
|
||||
machine.wait_until_succeeds(
|
||||
"journalctl -u earlyoom.service | grep -Eiq 'sending SIG(TERM|KILL)'",
|
||||
timeout=120,
|
||||
)
|
||||
# …the hog is gone…
|
||||
machine.wait_until_succeeds(
|
||||
'test "$(systemctl show -p ActiveState --value hog.service)" != "active"',
|
||||
timeout=60,
|
||||
)
|
||||
# …and the bystander is untouched.
|
||||
machine.succeed("systemctl is-active --quiet bystander.service")
|
||||
'';
|
||||
};
|
||||
|
||||
# The distroId question (roadmap "Distro branding"): set
|
||||
# distroId = "nomarchy" so /etc/os-release is honest (ID=nomarchy,
|
||||
# ID_LIKE=nixos). The risk is switch-to-configuration's "is this
|
||||
|
||||
@@ -22,7 +22,7 @@ let
|
||||
'';
|
||||
in
|
||||
{
|
||||
imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ];
|
||||
imports = [ ./options.nix ./plymouth.nix ./file-manager.nix ./power.nix ./services.nix ./hardware.nix ./timezone.nix ./oom.nix ];
|
||||
|
||||
config = {
|
||||
# Distro branding. distroName flows into /etc/os-release PRETTY_NAME,
|
||||
|
||||
42
modules/nixos/oom.nix
Normal file
42
modules/nixos/oom.nix
Normal file
@@ -0,0 +1,42 @@
|
||||
# Memory-pressure protection: keep the desktop alive when RAM runs out.
|
||||
#
|
||||
# A workstation that compiles from source WILL exhaust memory eventually —
|
||||
# a big `nix build`, a runaway eval, a browser tab. The kernel's own OOM
|
||||
# killer acts only after the system has thrashed itself unresponsive
|
||||
# (often minutes of frozen desktop) and then picks by badness score,
|
||||
# which can land on the compositor.
|
||||
#
|
||||
# earlyoom over systemd-oomd — a deliberate choice: oomd kills whole
|
||||
# cgroups, and a Hyprland session runs as ONE scope (nothing spawns
|
||||
# per-app systemd scopes here, unlike GNOME), so under pressure oomd
|
||||
# would take out the entire desktop to save it. earlyoom kills a single
|
||||
# process (highest oom_score ≈ the hog) BEFORE the thrash point — "kill
|
||||
# the build step, keep the session". nixpkgs default-enables oomd in an
|
||||
# inert state (no slices monitored); it's disabled outright below so
|
||||
# there is exactly one owner of the OOM story.
|
||||
{ lib, ... }:
|
||||
|
||||
{
|
||||
services.earlyoom = {
|
||||
enable = lib.mkDefault true;
|
||||
|
||||
# Desktop toast when something is killed (relayed via
|
||||
# systembus-notify), so a vanished build/tab is explained rather
|
||||
# than mysterious.
|
||||
enableNotifications = lib.mkDefault true;
|
||||
|
||||
# Never pick the session plumbing: losing the compositor or the lock
|
||||
# screen IS the outage this module exists to prevent (and killing a
|
||||
# Wayland session-lock client trips its go-to-a-tty failsafe). No
|
||||
# --prefer tuning: highest-memory selection already targets the hog.
|
||||
# Matched unanchored — NixOS wrappers rename comm to ".foo-wrapped".
|
||||
extraArgs = lib.mkDefault [
|
||||
"--avoid"
|
||||
"(Hyprland|hyprlock|greetd|waybar|pipewire|wireplumber|Xwayland|nix-daemon|systemd)"
|
||||
];
|
||||
};
|
||||
|
||||
# One owner (see header): oomd ships default-on but inert; make the
|
||||
# earlyoom choice explicit and total.
|
||||
systemd.oomd.enable = lib.mkDefault false;
|
||||
}
|
||||
Reference in New Issue
Block a user