Isn't programming also just telling the computer what to do? It gets converted to machine language, and you didn't write that yourself either.
The only difference I see is that programming uses more complex wording so it requires skill and thus can't be used by everyone.
It is like promting AI in Klingon.