If a phrase conveys the opposite of their literal meaning, and the speaker and the audience both know it, then it is pedantic. Choosing to derail whatever the topic is in favor of criticizing someone's understandability when everyone did understand them is pedantic.

I get it, I hate the way people use "literally". It's terrible, it's usually unneeded, and it just makes any actual correct use of literally have less impact. But I'm not gonna correct people who say it wrong, because I do know what they meant.

If they said "I could care less" and you're comfortable enough in your understanding of the conversation to know for a fact they actually mean they do not care about it, then they did make sense and you did understand them.