Because he was an (likely intelligence-affiliated) influence broker with a huge social circle. Doesn't make associating with him less scummy, but it does not mean someone was necessarily a pedophile. Though, statistically speaking, it does dramatically increase that probability.