Topsoil is worse than just erosion, it has no nutrients left. Without heavy use of petroleum based fertilizer, we'd be far, far worse off. And as it is our food is less nutritious than it was, not that long ago. But I don't see that as a civilization ender by itself but can and likely will contribute, it will also make it much more difficult to come back after the fall.