Timing could be a thing too. If done fast enough, they just see it as typical disruption, and finding everything as they left it go about living there. If it's gone for a day or more, then they see it as lost.
And they are protective, learned that with bluebirds and a bird house. But they also share some of the aspects of crows, and learn after a bit that you aren't a predator and are just one of the occupants in the area. They stopped buzzing me as long as I didn't linger in the spot or was nearby but doing other things.