Note · 2025-09-30 · desk machine, 6.6 kernel, schedutil throughout

The idle loop is not free

Choosing a frequency governor decides how fast a core runs while it has something to do. Underneath that sits a smaller and more interesting decision, taken thousands of times a second: when there is nothing to do, how deeply should the core be allowed to fall asleep, given that waking up takes time?

The shape of an idle period

A core that runs out of work does not drop straight into its deepest state. It walks down, and each step is cheaper to hold but slower to leave. The driver has to guess how long the quiet will last before it knows, which is the whole game.

What the states cost on this part

The target residency column is the honest part of the interface: it says «do not bother entering me unless you expect to stay longer than this». C6 wants 600 µs and pays back 133 µs of latency on the way out, and on a quiet machine it wins overwhelmingly - 92 % of the time, with a mean stay of 6.4 ms.

Now take it away

I ran the same ten-minute windows three times, changing only how deep the core is allowed to go:

A polling idle loop costs 3.8 W on a four-core desktop part, which is more than the entire difference between the best and the worst frequency governor in the previous note minus one watt. It is also invisible in every monitoring tool that only graphs processor utilisation, because a core spinning in the idle loop is, by that definition, idle.

People turn polling on for a reason - it removes the 133 µs and the jitter around it. If you are chasing a hard sub-millisecond deadline, pay the watts knowingly. If you inherited the setting from a wiki page written by somebody who was chasing one, take it back out.

Two ways the residency script lies

The counters live in /sys/devices/system/cpu/cpuN/cpuidle/stateN/, with usage as a count of entries and time as microseconds spent. Both are totals since boot, which produces the first lie: a single read of a machine that has been up for ninety days tells you about ninety days, not about now. Read twice, subtract, divide by the wall time between the reads. I have watched a very senior person declare a fleet healthy on the strength of one read.

The second lie is quieter. The time field does not include the sojourn that is still in progress, so on a short window with long stays you systematically undercount. The check that catches both mistakes is arithmetic:

sum(all state time deltas) + busy time
  should be within a percent or two of
  window seconds x number of cores

If that sum comes out at a third of the expected total, your script is reading one core and calling it the machine. If it comes out at three times, you are adding the per-core files twice because the topology lists each thread. Neither failure looks wrong on a graph; both look like a dramatic improvement.

The one derived number I actually use is time / usage - the mean stay. It is far more sensitive to a wakeup problem than residency is, because residency can stay high while the stays get chopped into useless fragments.

Update, 2026-09-18

That last sentence turned into the third note: a server with 91 % residency and a mean stay of 2.1 ms, which is the signature of something waking it up eighty times a second for no reason at all.