It's not "physically impossible" on Earth. The forces involved are great, sure, which means you can't build it out of any present-day material like steel, but they're not so great that constructing a space elevator would be physically impossible using non-exotic matter like it would be on, say, the Sun, or possibly even just Jupiter. We already know of materials that could be used to make a space elevator cable on Earth if they were available in sufficient quantities -- namely carbon nanotubes.
The "top" can't be anywhere, because not everywhere along the length of the elevator will put released objects in orbit. Turns out on Earth, an object released off of the elevator would reach a stable (but very eccentric) orbit 2/3rds of the way to geostationary orbit -- below that, it would fall back to Earth. Conversely escape velocity would be reached at about 53000 km, which is past geostationary orbit but much closer than where the counterweight would be (in most designs?). Objects above escape velocity will by definition escape Earth's orbit, which most of the time means ending up in a solar orbit.