Pretty sure ladybugs eat aphids, so why not have your cake and eat it too?

You've got it reversed; less insecticide = more ladybugs = less aphids. Ladybugs are their natural predator and can actually be bought from breeders along with stuff like tomato worm wasps as a natural non-chemical pest control.

I was also taught as a kid to only kill the tomato worms without eggs laid on them:

You let those ones live because the eggs are gonna hatch and eat it then make more wasps that will eat more tomato worms.

Source: grew up rural.