We actually don't know if integer factorization is not in P, though.
Right now, I think most of us would guess that it is a prime candidate for NP Intermediate. Hence why I mentioned it earlier.
And you absolutely can solve it in polynomial time just not with classical architectures.