feat(doctor): #147 — flag machine state the flake never set; linger is the worked example
Some checks failed
Check / eval (push) Has been cancelled
Some checks failed
Check / eval (push) Has been cancelled
/var/lib survives a migration untouched, and a stray pre-Nomarchy Linger=yes marker shaped the QA box's session behavior for six months with nothing in the flake to show for it. Ship the class as guards: - nomarchy-doctor: new machine-state-drift check after the flake-checkout check — loginctl show-user -p Linger; "yes" warns (nothing is broken — it is drift) and prints the disable-linger fix; unknown-to-logind skips. - docs/MIGRATION.md § Post-migration cleanup: "/var/lib machine state survives migration" — the worked example and the general suspicion rule. - checks.doctor: linger sub-case — warn fires under enable-linger, exit stays 0, warning clears after disable-linger. Deliberately NOT hardening the session wiring against the portal-gtk restart race linger enabled: a clean install cannot reach that state. The Waybar exec-once loose thread survives as BACKLOG #149. Verification: V2 — shellcheck clean, nix flake check --no-build green, nix build .#checks.x86_64-linux.doctor green (both branches exercised). Implementation by a Sonnet subagent; design and review on Fable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -265,64 +265,16 @@ If that count is zero over a few weeks post-linger, **close this unbuilt**. If i
|
|||||||
is not, the upstream issue has sat ten months on vaxerski's "that will cause a
|
is not, the upstream issue has sat ten months on vaxerski's "that will cause a
|
||||||
segfault no?" and a reproducer would be worth more than a local guard.
|
segfault no?" and a reproducer would be worth more than a local guard.
|
||||||
|
|
||||||
### 147. Nomarchy cannot see machine state it never set — and it bit the QA box for 6 months
|
### 149. `[watch]` Waybar's exec-once workaround may be defending against nothing post-linger
|
||||||
|
|
||||||
**Rewritten 2026-07-16 after diagnosis; the original filing was wrong and its
|
Spun out of #147 (shipped 2026-07-17) so its loose thread survives the entry:
|
||||||
wrongness is the point.** It claimed "a re-login leaves session units dead;
|
`modules/home/hyprland.nix` runs Waybar from `exec-once` rather than a systemd
|
||||||
the repo never adopted the standard session wiring". The repo *has* the wiring
|
unit because the unit "raced Hyprland's IPC on a **warm** relogin ... never
|
||||||
— HM's `wayland.windowManager.hyprland.systemd.enable` defaults on and is line
|
retried, so the bar vanished" — and a warm relogin is exactly what linger
|
||||||
1 of the generated `hyprland.conf` — and it fired correctly. Do not go add it.
|
created. With the stray linger removed (2026-07-16) that workaround may be
|
||||||
|
defending against nothing. Do not rip it out on this note alone: it works, and
|
||||||
**What actually happened**, from systemd's own words at the 17:25:54 relogin:
|
the claim needs a real relogin without linger to test (V3 — the same relogin
|
||||||
|
that verifies #148's watcher fix would answer it).
|
||||||
```
|
|
||||||
Stopped target Hyprland compositor session.
|
|
||||||
Requested transaction contradicts existing jobs: Transaction for
|
|
||||||
graphical-session.target/stop is destructive (xdg-desktop-portal-gtk.service
|
|
||||||
has 'start' job queued, but 'stop' is included in transaction).
|
|
||||||
graphical-session.target: Failed to enqueue stop job, ignoring
|
|
||||||
Reached target Hyprland compositor session.
|
|
||||||
```
|
|
||||||
|
|
||||||
`hyprland-session.target` stopped; `StopWhenUnneeded=yes` correctly tried to
|
|
||||||
take `graphical-session.target` with it; systemd **refused the stop** because
|
|
||||||
portal-gtk had a `start` job queued from its own post-crash restart loop (3
|
|
||||||
restart/fail lines in that 13s gap). So graphical-session never went down, 8ms
|
|
||||||
later hyprland-session came back to find it active, and nothing `WantedBy` it
|
|
||||||
was re-pulled. cliphist had already burned 5 restarts into `start-limit-hit`;
|
|
||||||
nm-applet/udiskie have no restart policy and just stayed dead.
|
|
||||||
|
|
||||||
**But none of that can happen on a Nomarchy install.** The whole sequence needs
|
|
||||||
the user manager to survive logout, i.e. `Linger=yes` — and linger came from an
|
|
||||||
empty marker in `/var/lib/systemd/linger/bernardo` dated **2026-01-24**, which
|
|
||||||
**pre-dates Bernardo's migration to Nomarchy** (he was on a custom flake then).
|
|
||||||
Nothing here sets it: not modules, not hosts, not templates, and `git log -S`
|
|
||||||
says it was never shipped, ever. Removed 2026-07-16 (`loginctl disable-linger`,
|
|
||||||
no restart needed); nothing wanted it — `nomarchy-updates.timer` is
|
|
||||||
`Persistent=yes` and catches up on login.
|
|
||||||
|
|
||||||
**So the real item is the class, not the bug.** `/var/lib` is machine state; a
|
|
||||||
migration reconciles the flake and leaves everything else exactly as it was.
|
|
||||||
Nomarchy therefore ran for six months on its own primary V3 box with a setting
|
|
||||||
it never chose, never set, and could not see — on the machine whose whole job is
|
|
||||||
telling us how the distro behaves. That is a direct hit on the in-flake-state
|
|
||||||
promise (state outside the checkout) *and* on V3's trustworthiness: a QA box
|
|
||||||
that silently diverges reports on a distro nobody ships.
|
|
||||||
|
|
||||||
**Pitch:** a `nomarchy-doctor` check for drift of this class — linger is the
|
|
||||||
worked example (`loginctl show-user $USER -p Linger` vs. what the flake asks
|
|
||||||
for), and MIGRATION.md should say plainly that `/var/lib` survives and what to
|
|
||||||
sweep. Cost is small and it is the only part of this worth shipping. **Do not**
|
|
||||||
harden the session wiring against the portal-gtk race: a clean install cannot
|
|
||||||
reach it, and defending against it would be clever code for a state we do not
|
|
||||||
ship.
|
|
||||||
|
|
||||||
**Loose thread for whoever takes this:** `modules/home/hyprland.nix` runs Waybar
|
|
||||||
from `exec-once` rather than a unit because the unit "raced Hyprland's IPC on a
|
|
||||||
**warm** relogin ... never retried, so the bar vanished" — and a warm relogin is
|
|
||||||
exactly what linger creates. That workaround may now be defending against
|
|
||||||
nothing. Do not rip it out on this note alone; it works, and the claim needs a
|
|
||||||
real relogin without linger to test.
|
|
||||||
|
|
||||||
### 120. A netinstall ISO, next to the fat offline one
|
### 120. A netinstall ISO, next to the fat offline one
|
||||||
|
|
||||||
|
|||||||
@@ -19,6 +19,21 @@ Template:
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-07-17 — #147: doctor learns to see machine state the flake never set
|
||||||
|
- **Task:** BACKLOG #147 (promoted from PROPOSED by Bernardo in-session).
|
||||||
|
- **Did:** `nomarchy-doctor` gains a machine-state-drift check — linger as the
|
||||||
|
worked example (`loginctl show-user -p Linger`; `yes` → warn + fix command,
|
||||||
|
unknown-to-logind → skip); MIGRATION.md § Post-migration cleanup gains
|
||||||
|
"`/var/lib` machine state survives migration"; `checks.doctor` grew the
|
||||||
|
warn-fires/warn-clears linger sub-case. Implementation delegated to a
|
||||||
|
Sonnet agent, design + review here. Loose thread preserved as #149.
|
||||||
|
- **Verified:** V2 — shellcheck clean, `nix flake check --no-build` green,
|
||||||
|
`nix build .#checks.x86_64-linux.doctor` green (linger sub-case exercised
|
||||||
|
both branches, doctor exit 0 throughout).
|
||||||
|
- **Pending:** nothing hardware-blocked; the real-world case (QA box linger)
|
||||||
|
was already fixed by hand 2026-07-16.
|
||||||
|
- **Next suggestion:** hardware hints, then #148 review.
|
||||||
|
|
||||||
## 2026-07-17 — Live triage with Bernardo: #114 ships as docs; rulings on #120/#134; #143 checked
|
## 2026-07-17 — Live triage with Bernardo: #114 ships as docs; rulings on #120/#134; #143 checked
|
||||||
- **Task:** Bernardo triaged PROPOSED in-session: #148/#147/hardware-hints
|
- **Task:** Bernardo triaged PROPOSED in-session: #148/#147/hardware-hints
|
||||||
promoted (in flight as separate commits), #120 re-affirmed deferred, #134
|
promoted (in flight as separate commits), #120 re-affirmed deferred, #134
|
||||||
|
|||||||
@@ -384,6 +384,26 @@ lock).
|
|||||||
prototype).
|
prototype).
|
||||||
- Prune old generations: `sudo nix-collect-garbage -d`.
|
- Prune old generations: `sudo nix-collect-garbage -d`.
|
||||||
|
|
||||||
|
### `/var/lib` machine state survives migration
|
||||||
|
|
||||||
|
A migration reconciles the **flake** — packages, services, dotfiles — and
|
||||||
|
nothing else. Everything under `/var/lib` is left exactly as the previous
|
||||||
|
OS wrote it, including settings the flake can neither see nor express. That
|
||||||
|
state doesn't show up in `git diff`, doesn't get touched by `sys-rebuild`,
|
||||||
|
and can sit there for months quietly shaping how the machine behaves.
|
||||||
|
|
||||||
|
Worked example: user lingering. Run `loginctl show-user $USER -p Linger` —
|
||||||
|
on a Nomarchy machine the answer should be `no`, because the flake never
|
||||||
|
sets it. On our own QA box a leftover `Linger=yes` from a pre‑Nomarchy setup
|
||||||
|
kept the user's systemd instance alive across logouts, and it took six
|
||||||
|
months to trace a string of subtle session bugs back to that one marker.
|
||||||
|
Fix: `sudo loginctl disable-linger <user>`.
|
||||||
|
|
||||||
|
`nomarchy-doctor` now flags this specific case as a warning. More broadly:
|
||||||
|
after migrating, if the machine ever behaves in a way the flake can't
|
||||||
|
explain, suspect inherited state under `/var/lib` before you suspect the
|
||||||
|
flake.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Enabling hibernation on an existing machine (no reinstall)
|
## Enabling hibernation on an existing machine (no reinstall)
|
||||||
|
|||||||
@@ -493,6 +493,17 @@ Design/decision records and a running log of shipped work (items marked
|
|||||||
V2 by render, both heads + the password dialog, via the new
|
V2 by render, both heads + the password dialog, via the new
|
||||||
`tools/plymouth-preview.sh`. V3 (a real docked boot/shutdown) is queued: the
|
`tools/plymouth-preview.sh`. V3 (a real docked boot/shutdown) is queued: the
|
||||||
fake heads exist from the start, so the rig cannot resize a canvas.
|
fake heads exist from the start, so the rig cannot resize a canvas.
|
||||||
|
- ✓ **Machine state the flake can't see (#147, 2026-07-17):** the QA box ran
|
||||||
|
six months with `Linger=yes` from a pre-migration marker in
|
||||||
|
`/var/lib/systemd/linger` — set by nothing in the repo, visible to no flake
|
||||||
|
audit. The class ships as guards, not session hardening: `nomarchy-doctor`
|
||||||
|
now warns on linger drift (`loginctl show-user -p Linger` vs. the flake's
|
||||||
|
implicit "no", fix command printed), and MIGRATION.md § Post-migration
|
||||||
|
cleanup states plainly that `/var/lib` survives a migration and what to
|
||||||
|
sweep. The portal-gtk restart race that linger enabled was deliberately NOT
|
||||||
|
hardened against — a clean install cannot reach that state. V2:
|
||||||
|
`checks.doctor` grew a linger sub-case (warn fires, exit stays 0, clears
|
||||||
|
after `disable-linger`). The Waybar exec-once loose thread became #149.
|
||||||
- ✓ **Dock mode survives a rebuild (#142, 2026-07-16):** Bernardo: every rebuild
|
- ✓ **Dock mode survives a rebuild (#142, 2026-07-16):** Bernardo: every rebuild
|
||||||
while docked re-lit the laptop panel and threw a workspace onto it, and he
|
while docked re-lit the laptop panel and threw a workspace onto it, and he
|
||||||
re-picked Dock mode by hand. It was not forgetfulness — **a rebuild performed
|
re-picked Dock mode by hand. It was not forgetfulness — **a rebuild performed
|
||||||
|
|||||||
12
flake.nix
12
flake.nix
@@ -1017,6 +1017,18 @@
|
|||||||
healthy = machine.succeed("nomarchy-doctor 2>&1")
|
healthy = machine.succeed("nomarchy-doctor 2>&1")
|
||||||
for probe in ["NetworkManager", "fingerprint", "fwupd", "VA-API", "hibernate", "battery health"]:
|
for probe in ["NetworkManager", "fingerprint", "fwupd", "VA-API", "hibernate", "battery health"]:
|
||||||
assert probe in healthy, f"doctor hardware section missing a self-gated '{probe}' row:\n{healthy}"
|
assert probe in healthy, f"doctor hardware section missing a self-gated '{probe}' row:\n{healthy}"
|
||||||
|
# Machine-state drift: linger (#147's worked example) — the
|
||||||
|
# flake never sets it, so a "yes" here is drift; doctor must
|
||||||
|
# warn (exit 0), not fail, and the warning must clear once the
|
||||||
|
# marker is gone.
|
||||||
|
machine.succeed("loginctl enable-linger root")
|
||||||
|
linger_on = machine.succeed("nomarchy-doctor 2>&1")
|
||||||
|
assert "Nomarchy never sets it" in linger_on, f"doctor did not flag linger drift:\n{linger_on}"
|
||||||
|
machine.succeed("loginctl disable-linger root")
|
||||||
|
linger_off = machine.succeed("nomarchy-doctor 2>&1")
|
||||||
|
assert "Nomarchy never sets it" not in linger_off, (
|
||||||
|
f"doctor still flags linger drift after disable-linger:\n{linger_off}"
|
||||||
|
)
|
||||||
# #80 battery health: faked sysfs (test_power has no cycle/design).
|
# #80 battery health: faked sysfs (test_power has no cycle/design).
|
||||||
machine.succeed(
|
machine.succeed(
|
||||||
"mkdir -p /tmp/fake-ps/BAT0 && "
|
"mkdir -p /tmp/fake-ps/BAT0 && "
|
||||||
|
|||||||
@@ -110,6 +110,19 @@ if [ -d "$flake/.git" ]; then
|
|||||||
fi
|
fi
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# ── machine-state drift (survives migration, unseen by the flake) ────
|
||||||
|
# Worked example: systemd linger. The flake never sets it anywhere, so
|
||||||
|
# a "yes" here is leftover state from a previous OS/install, not
|
||||||
|
# something Nomarchy configured — it can cause months of divergent
|
||||||
|
# session behavior that no amount of flake auditing will explain.
|
||||||
|
linger=$(loginctl show-user "${USER:-$(id -un)}" --property=Linger --value 2>/dev/null || true)
|
||||||
|
case "$linger" in
|
||||||
|
'') skip "user linger (user unknown to logind — no session)" ;;
|
||||||
|
yes) warn "user linger is enabled but Nomarchy never sets it (machine state the flake can't see)" \
|
||||||
|
"loginctl disable-linger ${USER:-$(id -un)}" ;;
|
||||||
|
*) ok "no user linger (matches the flake)" ;;
|
||||||
|
esac
|
||||||
|
|
||||||
# ── generation age ───────────────────────────────────────────────────
|
# ── generation age ───────────────────────────────────────────────────
|
||||||
# The profile SYMLINK's own mtime is the generation's creation time
|
# The profile SYMLINK's own mtime is the generation's creation time
|
||||||
# (store paths themselves are all epoch-1).
|
# (store paths themselves are all epoch-1).
|
||||||
|
|||||||
Reference in New Issue
Block a user