If they said "I could care less" and you're comfortable enough in your understanding of the conversation to know for a fact they actually mean they do not care about it

And what if I am not comfortable enough in my understanding? When someone is hard to understand because of how non-standard their use of language is, it is a communication barrier, not just pedantry.