it relies on P vs NP
That is incorrect. It relies on perceived difficulty of factoring large semi-prime. P equals NP is related, of course, if it happens to be an NP-intermediate problem, but the difficulty of factoring is not because it's NP-complete.
There are already classical algorithms that are sub-exponential to solve the problem. If it were the case that the problem was NP-complete, then we would have had a big advance in our theory and practice based on it.