You can break bones falling off even a stationary bike, yes, it matters. People step out in front of you at walking speed wearing headphones and cause crashes that result in not so minor injuries - especially if you're transferring your momentum into the pedestrian and their head hits the concrete. That's without even considering damage to bikes and equipment. Helmets need replacing after a single impact.
You're out of your mind if you think bikes are only really ridden by toddlers? And even then, what a great experience for toddlers to have preventable crashes