An offsite server is not under your control and accessible by who knows. Surely it is still a privacy concern.
Privacy is like security in that it costs time. Most people don't spend time on even having a conversation like this but if something bothers you then finding a spare day is easier.