Note · 2026-09-18 · shelf server, measured 5 to 14 September
Where 118 wakeups a second came from
The shelf server looked fine. Idle residency in the deepest state was 91.2 %, which is the number anybody would quote and nod at. The mean stay in that state was 2.1 ms, against 6.4 ms on a much cheaper desktop board doing the same amount of nothing. Something was chopping the quiet into pieces.
Accounting, not guessing
The temptation is to go straight to the suspects - the monitoring agent, obviously - change something, and see if the number moves. I have done that and it produces a story rather than an answer, because when two sources are responsible you fix one, see a third of the improvement, and conclude you were wrong about it.
So: count the total first, then attribute it, then check that the attribution adds up. Interrupt deltas per source from /proc/interrupts, read twice sixty seconds apart; the pending timer list; per-process wakeup counts over the same minute. The rule I hold myself to is that the sum of the sources must come within five per cent of the measured total, and until it does I have not found them all. The first pass accounted for 71 of 118 and was therefore worthless.
The five sources
| Source | Before | After |
|---|---|---|
| a daemon sitting in a 10 ms wait | 51 | 4 |
| monitoring agent, one-second interval, four threads | 38 | 5 |
| network card watchdog and link check | 12 | 12 |
| filesystem commit and journal flush | 9 | 9 |
| kernel timers with nothing to attribute them to | 8 | 8 |
| total | 118 | 38 |
The daemon was the surprise, and it is the same shape of mistake I used to make myself: a loop that waits ten milliseconds for an event that arrives perhaps forty times an hour. Raising that wait to 250 ms took one line and a restart.
The agent was no surprise at all. Four threads each asking the kernel for numbers once a second is not monitoring, it is a very slow stress test. I set its interval to ten seconds and gave it one thread.
The other three I left exactly as they were. The commit interval exists to bound how much I lose on a power cut, the card’s watchdog exists so a dead link gets noticed, and the eight unattributable timers are the kernel doing its job. Chasing those would be trading real properties for a number.
What it bought
| Measure | Before | After |
|---|---|---|
| idle draw at the wall | 33.6 W | 32.4 W |
| deepest-state residency | 91.2 % | 95.7 % |
| mean stay in that state | 2.1 ms | 7.9 ms |
| p99 of the one service on the box | 8.1 ms | 8.3 ms |
One point two watts. At the price I pay for electricity that is about two euros a year, so as an energy result this note is a waste of a fortnight and I want to be clear about that. The service latency did not improve either - 8.1 against 8.3 ms is the meter and the load generator arguing, not a change.
Why I still think it was worth doing
Because the mean-stay figure went from a number that cannot drift to a number that can. At 2.1 ms the metric was already pinned near the floor by my own software, so nothing short of a catastrophe could have moved it. At 7.9 ms there is room to fall, and a fall means something is waking the machine that was not waking it yesterday.
That is not a hypothetical. In 2022, on the fleet described in work, a node whose mean stay dropped from 6 ms to 1.4 ms over two days turned out to have a disk retrying reads; the disk’s own error counters were still clean, and the only thing that had noticed was the idle histogram. That is the whole argument for keeping a quiet machine genuinely quiet: it leaves you with an instrument instead of a flat line.
The trade I accepted, and you may not want to: monitoring at ten-second granularity cannot see a four-second stall. On a box with one service and me as the only user, that is fine. On something customer-facing it would not be.
Method and machine description are the same as in the other two notes, listed on the notes page. The earlier piece on idle states explains why mean stay is the number I watch.