I would be caring for the cat as an outside cat, I would wear long pants , forget the neighbours, they just do not care
first feed him up and get trust, then get drops to put in food
but mainly don't stress, he's a cat
set achievable goals, you don't have to look after him perfectly, good enough is good enough
and really, sometimes caring for someone else is a good way to destress