That's exactly why you want to do this. When people speak casually or are making jokes, nobody speaks with grammar in mind. Gathering this intuition for the language is exactly what I'm looking for when learning.

And a lot of native speakers just straight up use incorrect grammar. eg loads of native English speakers say things like "could of", use "then" instead of "than", use punctuation incorrectly, do incorrect sentence structure, etc.

I don't mean to be prescriptive about language. Of course it evolves based on usage. But in any formal environment, such as an academic environment, there are still certain defined and canonised English grammar rules that you would be penalised for breaking, and most learners want to learn those rules even if most casual speakers don't follow them.