I actually had to stop using Fluxer and just go to Steam's voice client. Even with noise suppression off and everyone in the same state the audio fidelity was seemingly getting worse every day. Waiting for when I can self host this but probably going to get bored and just setup xmpp instead.