What about planting something else that they'll eat like hop tree? Obviously that's not much of an immediate solution, though.
To answer your original question, it's always a balancing act. I love all kinds of butterflies and moths, but an asp on plants near my front door still had to go. They turn into the southern flannel moth, but as caterpillars are capable of a very painful sting.