Apparently Jupiter's gravity well can work both ways, either stopping asteroids from reaching us or giving them just enough energy to reach us. It's quite impartial at that. This video from Braintruffle covers the gravitational dynamics: https://youtu.be/1zu41rrc_Ng