If energy is movement, how come I get tired just from holding a heavy object?
(multiverse.soulism.net)
The distance is to the ground (technically to the centre of the nearest gravity well, but generally the ground gets in the way of that). The box would move towards the ground in order to expend potential kinetic energy and reach a lower energy state, but the counter force you're providing cancels it out, keeping it in place.