That was an argument made at the time for those who wished to 'let' slavery die a 'natural' death.

I think such a point misses that the North and the South were not one society or even one economic system. Slavery perpetuates itself not because it's economically efficient (it's not, generally) but because it empowers elites who then leverage that power to preserve that power. They won't let it go unless it's torn from them - the North had the advantage of having never really been a 'slave society'; slavery was permitted before it was banned, but slave-dependent elites were never a majority of power in the North.

It was always going to come down to coercion - whether through a prolonged struggle, or through the acquiescence of the slavers to an overwhelming threat of force. We had the former. I don't think the latter would have been possible without the South being a much smaller proportion of the country, economically, demographically, and geographically.