a large portion understands Swedish
Absolutely not the case, except for some Swedish speaking regions that are only a tiny part of the country and consist of villages and small towns.
Like 90% of Finns can't even have simple conversations in Swedish.
which would also be the reason why everyone would naturally hate me.
Why would "everyone" hate you?