Accumulating a history of chargebacks against you as a merchant, even if the consumer ultimately loses them, also counts against you and will raise your rates. The processors don't like dealing with merchants that they perceive as excessively risky.

I have to deal with this in my business and the whole thing is really a pain in the ass.