Hmm, you're right, that is definitely a problem. The best solutions I can think of would be-
-
Voter coercion laws. If nothing else make it at least illegal to do exactly that, so abusers can be held accountable.
-
Maybe limit who can receive delegations in some way. Like making the status of being a delegate carry responsibilities, requiring a small curriculum and civil service so that only people who genuinely care about being representatives pursue that path rather than any average asshole who might coerce their peers and family into giving them more personal voting power.