Sure. My parents had different religions and being an atheist I don't really have a duty to care about other people's religions.

Of course it helped that my parents weren't too seriously religious. And I've rejected religious people for having religion-tied views I find appalling. But the religion itself isn't the issue, just the things that sometimes result from it are.