From what I remember from college, Japan's ultranationalism in the 30s and 40s never really went away. I went to verify that and found a wiki page
And then anecdotally: my friend taught in Japan for 5 years and I experienced a bit of it myself. I also have a (white) friend who has lived there for 15+ years and has had plenty of unfriendly interactions - especially when he's with his Japanese wife or son.
I don't think it's unsafe to be a foreigner in Japan but I think it's fair to say Japan is more on the xenophobic side