I've met plenty of people who were smart in one very specific domain and dumb as bricks in pretty much all other areas. People around them would praise them from being smart but they just seemed ordinary or even below average outside their niche and I'd never refer to them as being smart myself.

Maybe you are being elitist by putting this guy you knew on a pedestal when outside that one skill he impressed you with he was actually a pretty dumb person.