My house had zero landscape plants when I bought it. Now it has several hundred. In a couple areas, I planted host plants just for caterpillars to eat, and am growing more from seed to have even more host plants. I'm even considering a native citrus specifically to attract swallowtail butterflies.
All that to say, your tree will be fine, even if it looks a little ragged.