I don't even think this is selfish. It's only selfish to put yourself first when you have kids who depend on you. People choose the responsibilities they take on in life, and you don't owe the world anything here. If you don't have kids, you're able to do more for yourself, and anyone who calls you selfish for it may actually be a bit jealous, in which case they should reassess their own parental priorities.