docs: RETRACT the #127 cause — the diagnosis was a measurement artifact; fix stands
All checks were successful
Check / eval (push) Successful in 3m30s
All checks were successful
Check / eval (push) Successful in 3m30s
The "blank + hypridle deadlock = the brick" cause published this morning is retracted everywhere it was asserted: ROADMAP, hyprland.nix's comment, #146, HARDWARE-QUEUE and two commit messages (a2151f4,321ccf7). It rested on two pieces of evidence and both were artifacts of how I measured: - "19 disconnects in 11 days": 18 of 19 are hypridle's noisy goodbye while systemd DELIBERATELY stops it — each bracketed by `Stopped hypridle` / `Started hypridle`, healthy replacement seconds later. - "Jul 14: CRITICAL at 13:08:17, then eleven hours of silence": an artifact of grepping ONE PID. systemd logged `Stopped hypridle` 13:08:17 and `Started hypridle` 13:09:00; PID 1678 ran fine all evening, invisible to the filter. Decisive: at the incident (before the 12:51:34 mitigation commit) hypridle had NO disconnect at all and had been healthy since Jul 14 13:09. It should have fired on-resume normally. The deadlock theory never had a case. The FIX stands on its own evidence and is untouched: the options really were off, hypridle's on-resume really was the only caller of `dpms on`, and input now wakes the screen with hypridle deliberately stopped (4s docked / 6s clamshell). Removing a proven single point of failure never needed a story about why the daemon might fail — which is exactly why it survived the story being wrong. #127 is RESTORED to NOW (deleted this morning as "shipped"; only the fix shipped). Its cause is open and the lead is the dead Ctrl+Alt+F3 — neither DPMS-off nor a dead hypridle can break VT switching, so something was wedged below the compositor holding DRM master. Every theory so far has been about the wake path, i.e. a symptom of the screen already being black. #146 → [watch], near-dead: one genuine zombie ever (verified by ps/ss, not grep); upstream #171 is real but its impact path closed when #147's stray linger went. Re-measure properly before building anything; close unbuilt if zero. Also closes the #127 hardware-queue entry: Bernardo ran the real 10-min idle — 19:27:58 dpms off (genuine timeout, shipped config, and it BLANKED = the revert working) -> mouse -> 19:32:14 `done enabled=1 first=DP-2`, clamshell, the incident geometry, one healthy hypridle throughout. All 4 steps pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
110
agent/BACKLOG.md
110
agent/BACKLOG.md
@@ -31,6 +31,44 @@ post-boot sessions). Preserve separation: the installer bake failure and the
|
||||
flake pin are different root causes even when they show up on the same machine.
|
||||
(Terminal / Ghostty-on-Acer → shipped as Kitty-only, #95.)
|
||||
|
||||
### 127. `[blocked:hw]` The idle brick's CAUSE — reopened; the fix shipped, the diagnosis did not
|
||||
|
||||
**Restored 2026-07-16, same day it was deleted.** It was deleted as shipped on
|
||||
the strength of a diagnosis that was retracted hours later. The **fix** is real,
|
||||
proven and stays (`misc:{key_press,mouse_move}_enables_dpms`, a2151f4): the wake
|
||||
path had a genuine single point of failure — hypridle's `on-resume` was the only
|
||||
caller of `dpms on` in the session — and it is gone. Verified end-to-end on the
|
||||
incident hardware, including a real 10-min idle blanking and waking clamshell
|
||||
(`done enabled=1 first=DP-2`, 19:32). **None of that explains the brick.**
|
||||
|
||||
**What the incident was:** docked clamshell, idle → black, external keyboard and
|
||||
mouse woke nothing, unplugging the external did not light the panel, Caps Lock
|
||||
LED still toggled (machine alive), and **Ctrl+Alt+F3 gave no usable TTY**.
|
||||
|
||||
**What has been ruled OUT (all on hardware, 2026-07-16):**
|
||||
- *DPMS-off is not the trigger.* With hypridle driving the blank, input wakes it
|
||||
in every geometry — undocked lid-open, clamshell-docked on the sole live
|
||||
output, docked with both outputs live. The clamshell skip was reverted on this.
|
||||
- *The hypridle deadlock is not the trigger.* This was the published cause and it
|
||||
was wrong: at the incident (before 12:51:34 on Jul 15) hypridle had **no
|
||||
disconnect at all** and had been healthy since Jul 14 13:09. The "19 in 11
|
||||
days" rate that made the theory look strong was normal-teardown noise
|
||||
(see #146). It should have fired `on-resume`.
|
||||
- *The #100 undock latch* explains symptom 2 (unplugging not restoring the panel)
|
||||
and is fixed (ff5017e) — but not the blackout itself.
|
||||
|
||||
**The lead is Ctrl+Alt+F3, and it always was.** Neither DPMS-off nor a dead
|
||||
hypridle can break VT switching; a wedged compositor still holding DRM master
|
||||
can, and it would explain the black screen *and* the dead TTY in one stroke.
|
||||
Every theory so far has been about the wake path — i.e. about a *symptom of the
|
||||
screen being black* — and none of them touch the TTY. Start there instead.
|
||||
Bernardo does not recall whether the machine had been relogged-in between Jul 14
|
||||
13:09 and the incident; that would say whether hypridle was ever a suspect.
|
||||
|
||||
**Note the fix makes this much harder to reproduce** (input now wakes the screen
|
||||
compositor-side), which is a good trade but means a repro may need the options
|
||||
temporarily off. It has not recurred since.
|
||||
|
||||
## NEXT
|
||||
|
||||
### 143. Rofi keeps the selected *row number* when the search changes, so the highlight lands on an unrelated entry
|
||||
@@ -165,48 +203,44 @@ settles. Note the ordering trap — the reload is also the undock rescue, so
|
||||
whatever re-asserts must not fight the rescue that deliberately re-enables eDP
|
||||
when the lid is open.
|
||||
|
||||
### 146. hypridle dies silently and systemd calls it healthy
|
||||
|
||||
Fell out of the #127 diagnosis (2026-07-16, ROADMAP § "#127 solved"), and it is
|
||||
the bug that was actually behind the brick — the DPMS gate we just reverted was
|
||||
only ever a symptom shield. hypridle hits an unfixed upstream deadlock
|
||||
(hyprwm/hypridle#171, "Disconnected from pollfd id 1" — open since 2025-09, in
|
||||
neither 0.1.7 nor main; upstream's own reporter saw 10-hour hangs) and then
|
||||
**hangs without exiting**, so `Restart=always` never fires and `systemctl
|
||||
--user is-active` still says `active (running)`. On the dev box: **19 times in
|
||||
11 days**, including the incident day. Jul 14 is the clean example — locking
|
||||
normally at 13:02, `CRITICAL` at 13:08:17, then eleven hours of silence, with
|
||||
D-Bus receive queues visibly backing up (48 KB unread) because nothing was
|
||||
reading them. The compositor is *fine* through all of it (no other session unit
|
||||
dies in the same second), so this is not the compositor's death taking it down.
|
||||
### 146. `[watch]` hypridle hangs instead of exiting when its compositor vanishes
|
||||
|
||||
Now that input wakes the screen compositor-side this is no longer brick-class,
|
||||
but it is not cosmetic either: a silently dead hypridle means **no auto-lock**
|
||||
(Bernardo sat unlocked for 25 min on 2026-07-16 without knowing), no blank, no
|
||||
idle suspend. Cost: systemd can't see it — `WatchdogSec` needs `sd_notify`
|
||||
hypridle doesn't implement, and the process never exits — so detection has to be
|
||||
external: a guard watching for the `CRITICAL` line, or a liveness probe on its
|
||||
Wayland fd, then `systemctl --user restart hypridle`.
|
||||
**Rewritten 2026-07-16, hours after filing: the premise was disproven and almost
|
||||
nothing survives.** As filed it said hypridle "dies silently ~2×/week — 19 times
|
||||
in 11 days", with Jul 14 as the clean example ("`CRITICAL` at 13:08:17, then
|
||||
eleven hours of silence"). Every number there was an artifact of the
|
||||
measurement. **18 of the 19** `Disconnected from pollfd id 1` lines are
|
||||
hypridle's noisy goodbye while systemd *deliberately stops it* — each bracketed
|
||||
by `Stopped hypridle` / `Started hypridle` with a healthy replacement seconds
|
||||
later. The "eleven hours of silence" came from grepping **one PID**: systemd
|
||||
logged `Stopped hypridle` 13:08:17, `Started hypridle` 13:09:00, and PID 1678
|
||||
ran fine all evening where the filter could not see it. Do not re-derive the old
|
||||
rate; it counts normal shutdown noise.
|
||||
|
||||
**Correction 2026-07-16:** this entry originally said "sequence it after #147,
|
||||
because restarting hypridle into a dead compositor fail-loops into
|
||||
`start-limit-hit` exactly as cliphist does". **That reasoning is void** — #147
|
||||
turned out not to be a distro bug at all (it needed stray pre-migration linger,
|
||||
since removed), and cliphist's fail-loop was a symptom of that same stray state.
|
||||
There is no ordering constraint: **#146 stands on its own and can be taken
|
||||
first.** The underlying caution survives in weaker form and is still worth
|
||||
designing for — a guard must not restart hypridle into a session whose
|
||||
compositor is gone — but that is ordinary care in the guard, not a dependency
|
||||
on another item.
|
||||
**What is real, and it is small.** Exactly one genuine zombie has been observed
|
||||
(2026-07-16 17:25:39), verified by process inspection rather than logs — `ps`
|
||||
(alive 9h36m), `ss` (48 KB unread on its D-Bus socket, i.e. not reading),
|
||||
`NRestarts=0` while `systemctl --user is-active` said `active (running)`. That
|
||||
is upstream hyprwm/hypridle#171 ("Possible deadlock in pollThr when compositor
|
||||
exits?", open since 2025-09, in neither 0.1.7 nor main) and it is a true bug:
|
||||
hypridle hangs rather than exits, so `Restart=always` never fires.
|
||||
|
||||
The trigger is still unknown — *not* dock/undock (no display-watch activity in
|
||||
the 3 min before any of them) and *not* a D-Bus quota disconnect (no broker
|
||||
errors). **Note the 19-in-11-days figure was measured on a box with linger on**;
|
||||
whether the rate holds now that the user manager is torn down at logout is
|
||||
itself unknown, and re-measuring is the cheap first step before building a guard.
|
||||
Worth a reproducer + an upstream comment on #171: it has sat ten months on
|
||||
vaxerski's "that will cause a segfault no?", and we have better data than the
|
||||
reporter did.
|
||||
**But its impact path is closed, which is why this is `[watch]` and not work.**
|
||||
The zombie only *persists* if systemd never stops the unit — which needed the
|
||||
user manager to survive logout, i.e. the stray pre-migration linger removed in
|
||||
#147. Without linger, logout tears the manager down and takes any zombie with
|
||||
it; the Jul 14 trace above shows the healthy path (`Stopped` → `Started`) doing
|
||||
exactly that. A Hyprland death ends the logind session anyway (2026-07-16:
|
||||
`Session 2 logged out` → greeter), so there is no known route left to a
|
||||
surviving zombie.
|
||||
|
||||
**Do before building anything:** re-measure. Count only disconnects with **no**
|
||||
`Stopped/Started hypridle` around them and **no** logind session event, and
|
||||
confirm by `ps`/`ss` that the process is really wedged — not by grepping a PID.
|
||||
If that count is zero over a few weeks post-linger, **close this unbuilt**. If it
|
||||
is not, the upstream issue has sat ten months on vaxerski's "that will cause a
|
||||
segfault no?" and a reproducer would be worth more than a local guard.
|
||||
|
||||
### 147. Nomarchy cannot see machine state it never set — and it bit the QA box for 6 months
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@ Run these first when you have a laptop session (AMD dev box unless noted):
|
||||
| # | Item | Why first |
|
||||
|---|------|-----------|
|
||||
| 1 | **#115** suspend-then-hibernate | Just shipped; quick path is minutes (battery vs AC + `nomarchy-suspend`) |
|
||||
| 2 | **#127** fix from config (not runtime keyword) | Cause found + fixed 2026-07-16; confirm our Nix emits it and it survives a relogin |
|
||||
| 2 | **#127** the brick's *cause* (reopened) | Wake path fixed + fully verified 2026-07-16; cause unknown — start at the dead Ctrl+Alt+F3, not the wake path |
|
||||
| 3 | **Docking recovery round 8** + headphone jack-follow | Same dock session; relogin once, then undock ×5 + jack |
|
||||
| 4 | **#104** airplane mode | Fast radio smoke after `nomarchy-home` |
|
||||
| 5 | **#101** charge-limit USB-C burst | Dock plug/unplug storm; pairs with #3 |
|
||||
@@ -54,14 +54,17 @@ Everything else below stays open; order is convenience, not a gate.
|
||||
blank, input woke it in all three configurations including clamshell-docked
|
||||
on the sole live output. The mitigation was reverted and #135 retired
|
||||
unbuilt, which also voids TEST A (its premise — the skip — no longer
|
||||
exists). Real cause: Hyprland's input→DPMS options ship off, so hypridle's
|
||||
`on-resume` was the only caller of `dpms on`, and hypridle deadlocks
|
||||
(hyprwm/hypridle#171) without exiting. Full record: ROADMAP § "#127
|
||||
solved"; the surviving bug is BACKLOG #146. **Method note worth keeping:**
|
||||
exists). What that leaves: Hyprland's input→DPMS options ship off, so
|
||||
hypridle's `on-resume` was the only caller of `dpms on` — a real single
|
||||
point of failure, now removed. It is **not** known to be what bricked the
|
||||
machine: the "hypridle deadlock" cause published on the day was retracted
|
||||
the same day (hypridle had no disconnect at the incident). #127 is reopened
|
||||
for its cause; ROADMAP § "#127's WAKE PATH fixed" is the full record.
|
||||
**Method note worth keeping:**
|
||||
the *first* TEST B was invalid — it ran `dpms off` by hand, which no wake
|
||||
path watches, so its "input did not wake it" was guaranteed regardless of
|
||||
hardware. Drive the real code path or measure nothing.
|
||||
- [ ] **#127 fix from config — steps 1+4 PASSED 2026-07-16, 2+3 open (AMD dev box)**
|
||||
- [x] **#127 fix from config — ALL 4 STEPS PASSED 2026-07-16 (AMD dev box)**
|
||||
— a2151f4 applied via `nomarchy-pull` + `nomarchy-home` on the dev box.
|
||||
**Done, no need to redo:**
|
||||
1. ✓ Both options read `int: 1` after `hyprctl reload` (which discards
|
||||
@@ -83,16 +86,20 @@ Everything else below stays open; order is convenience, not a gate.
|
||||
of anything: `/proc/acpi/button/lid/*/state` is, and it was never
|
||||
checked. Docked-lid-open remains **untested** — moot now that the gate
|
||||
is gone, but not tested. Do not read output lists as lid positions.
|
||||
**Still open — needs real idle time, cannot be rushed:**
|
||||
2. Docked, **lid closed**, idle 10+ min → the external **blanks**. This is
|
||||
the revert working; nothing to date has watched a genuine 10-min
|
||||
hypridle timeout fire, only forced blanks.
|
||||
3. Then press a key → it returns, and `journalctl --user -t
|
||||
nomarchy-display-wake --since '-15 min'` has a `done enabled=…` line
|
||||
(i.e. hypridle's on-resume ran *as well as* the compositor waking it).
|
||||
2+3. ✓ **The real idle path, no scaffolding** (Bernardo, 19:22–19:32,
|
||||
docked **lid closed**): `19:22:58 Executing pidof hyprlock || hyprlock`
|
||||
(5 min) → `19:27:58 Executing hyprctl dispatch dpms off` — the genuine
|
||||
10-min timeout firing from the shipped config, and it **blanked**, which
|
||||
is the revert working → mouse move → `19:32:13 Resumed: … Executing
|
||||
…/nomarchy-display-wake` → `19:32:14 done enabled=1 first=DP-2`.
|
||||
`enabled=1 first=DP-2` = clamshell, the incident's exact geometry. One
|
||||
healthy hypridle (PID 683145) throughout.
|
||||
Note: step 1 used `hyprctl reload` rather than a full relogin — a reload
|
||||
re-reads the same on-disk config a new session would, so the gap is
|
||||
cosmetic, but a relogin has not literally been done.
|
||||
**Entry closed. #127 itself is reopened for its CAUSE (see BACKLOG) —
|
||||
the wake path is fixed and verified; what bricked the machine is not
|
||||
known, and the dead Ctrl+Alt+F3 is the lead.**
|
||||
- [ ] **#96 battery-limit row on threshold-less firmware (Acer M5-481T)** —
|
||||
on the Acer (live or installed session with the commit carrying this
|
||||
entry): open System (`SUPER+CTRL+I`). **Pass:** a "Battery limit" row
|
||||
|
||||
@@ -19,7 +19,58 @@ Template:
|
||||
|
||||
---
|
||||
|
||||
## 2026-07-16 — RETRACTION: the #127 diagnosis was as wrong as the test it replaced
|
||||
- **Task:** Bernardo said "do #147 first, then #146". Both dissolved under
|
||||
investigation, and taking them apart disproved the #127 *cause* I had shipped
|
||||
hours earlier. The fix stands; the story around it did not. Sweep = retract.
|
||||
- **The retraction, and it is the whole entry.** I published "blank + hypridle
|
||||
deadlock = the brick" into ROADMAP, hyprland.nix's comment, #146 and two commit
|
||||
messages, on two pieces of evidence: "19 disconnects in 11 days" and Jul 14's
|
||||
"eleven hours of silence". **Both were artifacts of my own measurement.** 18 of
|
||||
19 disconnects are hypridle's goodbye while systemd *deliberately stops it*
|
||||
(each bracketed by `Stopped`/`Started`, healthy replacement seconds later). The
|
||||
"silence" was a grep for **one PID** — systemd logged `Started hypridle` 43s
|
||||
later and PID 1678 ran fine all evening, invisible to my filter. **Decisive:**
|
||||
at the incident hypridle had **no disconnect at all**, healthy since Jul 14
|
||||
13:09. The theory never had a case.
|
||||
- **The irony is the lesson.** I spent the morning writing up "the invalid TEST B
|
||||
measured its own scaffolding — put the mechanism inside the circuit", then did
|
||||
exactly that twice more: inferred a lid position from an output list, inferred
|
||||
"the session wiring is missing" from a grep that never covered HM's generated
|
||||
conf. Three wrong claims, all from inferring where checking was cheap; Bernardo
|
||||
caught all three. **Verify the instrument before believing the reading —
|
||||
hardest when the reading agrees with you.**
|
||||
- **#147 → not a distro bug.** The wiring exists (HM's `systemd.enable`, line 1 of
|
||||
the generated conf) and fired; systemd *refused* the `graphical-session.target`
|
||||
stop because portal-gtk had a start job queued ("destructive transaction"). It
|
||||
needs `Linger=yes` to matter — an empty marker dated **2026-01-24, pre-dating
|
||||
Bernardo's migration to Nomarchy**. Never shipped by us, ever. Removed. The item
|
||||
is rewritten to the class that IS real: `/var/lib` survives a migration, so the
|
||||
primary V3 box ran 6 months on a setting the distro never chose and cannot see.
|
||||
- **#146 → `[watch]`, near-dead.** One genuine zombie ever (verified by `ps`/`ss`,
|
||||
not grep); upstream #171 is real but its impact path closed with linger.
|
||||
- **Did:** retracted the cause in place everywhere it was asserted; **restored
|
||||
#127** (deleted this morning as "shipped" — only the fix shipped); rewrote #146
|
||||
and #147; filed #148 (a reload re-lights the panel in a shut lid — found because
|
||||
my own `hyprctl reload` did it to Bernardo, twice).
|
||||
- **Verified: V3, the fix survives all of it.** The real 10-min idle, no
|
||||
scaffolding: `19:27:58 Executing hyprctl dispatch dpms off` → blanked (the
|
||||
revert working) → mouse → `19:32:14 done enabled=1 first=DP-2` — clamshell, the
|
||||
incident geometry, from shipped config. Plus the brick condition on purpose
|
||||
(hypridle stopped): keypress woke it 4s docked / 6s clamshell. **The fix needed
|
||||
no story about why the daemon fails — which is why it survived the story being
|
||||
wrong.** V1: flake check + option-docs/clamshell-logind/docking-ux green.
|
||||
- **Next suggestion:** #127's **cause**, and start at the **dead Ctrl+Alt+F3** —
|
||||
the one symptom no theory yet has touched. DPMS-off cannot break VT switching
|
||||
and neither can a dead hypridle; a compositor wedged holding DRM master
|
||||
explains the black screen and the dead TTY together. Every theory so far has
|
||||
been about the wake path, i.e. a symptom of the screen already being black.
|
||||
|
||||
## 2026-07-16 — #127 SOLVED: the brick was never DPMS, and TEST B's first run lied
|
||||
- **⚠ SUPERSEDED — read the entry above first.** Its headline ("SOLVED") and its
|
||||
cause ("hypridle deadlock", "19× in 11 days") were **retracted the same day**:
|
||||
the evidence was a measurement artifact and hypridle had no disconnect at the
|
||||
incident. What survives: the invalid-TEST-B analysis, and the fix itself.
|
||||
- **Task:** #127 TEST B (Bernardo ran it), then the whole sweep his result forced.
|
||||
- **Read this if nothing else — the test measured nothing, and said so.** His run
|
||||
reported "input never wakes it" **docked *and* undocked alike**, and that tie is
|
||||
|
||||
Reference in New Issue
Block a user