irks me that it’s mostly idling
Well it's a small processor and relatively efficient one at that so... how about going the opposite direction? How about measuring the power draw on idle? With other task? I don't actually know if that architecture handles that but I saw some things on the do https://developer.arm.com/documentation/100095/0002/functional-description/power-management/dynamic-power-management?lang=en
Also what about using a RPi Zero instead?