Actually it's sort of scientific consensus. Not that it matters much for casual conversation (do not fuck minors, what's so hard about that), but the distinction is real.

obligatory nerdy note that all those -philias refer to attraction, and not action; pedophile might not abuse children but still be pedophile; child abuser is not necessarily pedophile