Factoring is not NP-complete because it is not a member of NP-hard.
You are correct that verifiability is in there, but you have to also be able to do the reduction.
There are plenty of problems out there where we lack polynomial time solutions, while the problem also lacks the expressiveness required to reduce back and forth.