Only if you make the assumption of a unimodal distribution of voters does that logic hold.
And that assumption clearly hasn't held. It's a convenient but wrong assumption that voters follow a left to right distribution centered on a mean. But it's not realistic and never has been.
People are complex and their motivations are complex, which results in a peaky, multidimensional distribution along separate often orthogonal axis.
If you try and target a "center of mass" in this scenario, you'll often hit a void of policy positions no one particularly likes, empty space.
It's like if 50 percent of voter want to go to Brussels, and 50 percent want to go to Paris and you propose Rancourt as a compromise solution, you get no one's vote because no one asked for that.