Not at all, society just needs to learn how to hold people to account themselves. It's a new way of thinking from what the majority are used to, but it's possible.
I've been in anarchist syndicates before and one of the major hurdles was preventing the bystander effect. Getting people to act of their own volition instead of waiting for the "adults" to step in. It's difficult, because it's new and foreign to most people, but it's certainly doable.
Many eyes and ears, all with the authority to call out bad actors, is far better than one central authority that you must go running to and hope that they believe you, and aren't themselves bad actors.
Want proof? Look at any large western country. All of them have vast swathes of people suffering due to bad actors, and all of them have a strong central state authority. How are the bad actors under control?