Sudden large population changes can cause problems. Everyone seems to inherently understand the issue with rapid growth, but I often see people saying rapid decline is totally fine and sometimes even a good thing, actually.

Every job suddenly being understaffed would be a challenge. Every piece of infrastructure suddenly being oversized creates inefficiencies. National pensions take money from workers and give it to the elderly, which only works if workers far outnumber the elderly. Some countries are headed to a 1:1 ratio and China is estimated to soon have 5 dependents for every 1 worker. Pension plans will go bankrupt.

All of this will happen all at once, everywhere, in a few decades. I don't think it's going to create apocalyptic scenarios, but people were freaking out over gas prices being twice as expensive, imagine what they're going to do when their pension payments stop.