There is technically a difference in the definition, but mostly people use it exactly as you're describing.

I've really had to catch myself when I notice myself using it.

But honestly it's so expected that people can get confused when you call yourself an immigrant (and you aren't doing it to make yourself a martyr somehow).