Wifi tracking uses measurements of the interference caused when objects pass between connected devices and a router. For your neighbors to track your movements their router would have to be connected to a device in your house, or maybe in the other neighbor's house on the far side of your house, so when you walk around it interferes with that signal.