I agree with you. Successful people often don't want to recognise how much good luck they've had. Childhood privileges that others don't have, job opportunities that others haven't had, etc.
If we applied their thinking to every situation, we'd say something like "if a child in Gaza dies from an Israeli bomb, it's the child's fault, because they didn't try hard enough to avoid the bomb".
Even if a right-winger admits that children can have different circumstances, they might have an attitude of "once you hit 18, we're all equal, and your effort is what determines your success in adult life". But that's not true. Some adults have a lot of privilege behind them, while others don't. Some adults have health conditions which can be very difficult to manage in some cases, taking up time and money and effort, while other adults don't have to worry as much about their health.