What I do is: Up, Home, type cd , Delete (hold/repeat as many times as needed), Enter.

The Home key instantly brings your cursor to the start of the command line (before the ls), even if the command is long enough to wrap.

I'm not a terminal guru, and I don't aspire to become one. I don't want to memorize arcane symbols that each has a subtly different meaning, and I especially don't want to execute them before I see explicitly what they're going to do. So while the top comment of cd !$ is a perfectly good and correct answer, I offer this alternative in case you, like me, like to see what you're doing as you do it. Home and End are pretty good.