I wasn't allergic for many, many years - at least, I assume I was exposed to it over those decades, but I never got it. But, in the last two years, I've gotten it twice.
So, I would say: Be very careful even though you aren't currently allergic.
Also, for anyone who has gotten it, this stuff (or equivalent) seems to work pretty well, as long as you do it right. It doesn't clear it up immediately, but it definitely knocks it down some. It's not cheap, but poison ivy irritation sucks.