If you want to listen to their mic via bluetooth or whatever, yes. But there's also this:
Some devices also support Google's Find Hub network. This enables users to find their lost accessories using crowdsourced location reports from other Android devices. However, if an accessory has never been paired with an Android device, an attacker can add the accessory using their own Google account. This allows the attacker to track the user via the compromised accessory.