To my understanding, people with this issue use the term to differentiate themselves (i.e. attracted to minors, but haven't acted on those impulses) from those who have acted on said impulses (pedophiles). Given the huge stigma connected to the label, I can perfectly understand why they'd want to make that distinction very clear.